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ABSTRACT 

This first volume of a two-volume report describes a 
project at the New Jersey Institute of Technology (NJIT) which 
assessed the effectiveness of a Virtual Classroom (VC) in which 
students and teachers communicate through a computer-mediated system 
called the Electronic Information Exchange system (EIES) . Chapter 1 
provides background on project goals, learning in the VC, educational 
technology and ef f ejtiveness , software, a theoretical framework, and 
outcon.'^^s to be measured. A discussion of methodology, covering target 
courses and subjects, experimental design, evaluatio*^., measurement, 
and data analysis is presented in Chapter 2. The next chapter deals 
w'th implementation problems related to student recruiting, 
equipnieut, software, resistance to collaborative learning, electronic 
pranks, and experimental controls. Chapter 4 describes student 
perceptions of the VC based on pre- and post-course questionnaires. 
Differences in course outcomes as affected by mode of delivery 
(completely online, mixed, or face-to-face) are discussed j.n Chapter 
5, while Chapter 6 looks at the effects of student attitudes, 
attributes, behavior, c.nd access conditions on outcomes. Findings are 
summarized in the final chapter Appendixes include: (1) the baseline 
questionnaire for students, with frequency distributions; (2) the 
post-course questionnaire for students, with frequency distributions; 
(3) the questionnaire for students who dropped the course, with 
frequency distributions; (4) the guide for interviews with students; 
and (5) interview transcripts. (90 references) (HES) 
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Foreword and Acknowledgments 

Many people who would like to attend college are unable to do so 
becausie they haven't the time or means to get to traditional 
classrooms on a traditional schedule. The person with a career 
outside the home, the person caring for small children, the disabled 
person - all of these individuals may find themselves shut out from 
furthering their education. 

Other students find the traditional classroom to be boring or 
ineffective for them. For instance, they might like to play a more 
active role in discussions and projects applying the skills and ideas 
covered in the courses, or to have more control over the pace a 
which material is covered. 

The Virtual Classroom, an innovative program originating at New 
Jersey Institute of Technology, brings the university into the homes 
and work places of such students through the use of computers. 
Specially designed computer software electronically links the Virtual 
Classroom titudent to his or her professors and classmates. Using a 
microcomputer, a telephone, and a device called a moder, the student 
attends lectures, takes tests, receives feedback from professors, 
attends conferences with fellow students, and more. The advantage is 
that the student need not adhere to a schedule of class meetings. 
The student decides at what time of day he or she will review a 
lecture, afik a professor a question, take a test, etc. Computer 
messages can be sent by the student and the professor at any time of 
the day or night. 

During the second year of the project, "Tools for the 
Enhancement and Evaluation of a virtual Classroom," prototypes of 
software tools to support online classes were implemented within 



"EIESl," the Perkin-Elme— based version of the Electronic Information 
Exchange System, and courses were conducted partially and totally 
online, m addition, during this time work progressed on PC-based 
software, called "Personal TEIES," which allows the integration of 
graphics (pictures, equations, and other symbols not present on a 
standard keyboard) with text. As an operational trial of a new mode 
of educational delivery, a variety of evaluation methods were used to 
assess the effectiveness of the Virtual Classroom, especially as 
compared with courses taught within a traditional (physical) 
classroom, of particular interest was the identification of 
variables which were related to relatively good and relatively poor 
outcomes for students within this new educational environment. This 
report of results is divided into two parts; Volume i includes a 
project overview and results from the students' points of view, and 
Volume 2 presents the experiences of the instructors and a guide for 
effective teaching online. Volume 1 incorporates extensive material 
from two interim reports: 

^^TisH^^^^llf^.^i^^ ^ Foundations. Research Report 
^4, cccc at NJIT, September 1986. 

.EY|iMtin3 |he Yir|Hal Classroom; Revised and Ugdited Plan, cccc 
Technical Report 87-16, March isaT. " , ^^^^ 

Detailed specifications for the software appear separately: 
^^and Funcg?nn^i^^n4o ?^^''!?-"'^ Capabilities in Conferences: A Manual 

'•R;port%'7-i27li^^ Tools on EIES. Technical 

'^''SLf Design Specifications for Personal TEIES 2.0: 
^fg G raphics Composition System and Personal CoiEIKIci^ns 
Manager. Technical Report 87-15.2, CCCC at NJIT, 1987: 

"4?^."^^^^"^' M^llMM for Personal TEIES 1.0. Technical ReDort 
86-4, CCCC at NJIT, 1986 (Revised 1987)": " ^nnicdi Keport 

During the third year of the project, the software tools 
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designed and implemented on EIESl will be rewritten in the '"C" 
language and implemented on TEIES, the Tailorable Electronic 
Information Exchange System. A Virtual classroom on TEIES will 
operate on any IBM-VM mainframe, and will be made available for lease 
to interested educational institutions. Limited beta testing will be 
carried out, but no systematic evaluation such as reported here will 
be conducted, unless additional funding is secured. 

In "Building the Foundations," I described my role as Principal 
Investigator for this project as something like tiat of an orchestra 
conductor. I had a vision of what the final product should be like. 
To achieve it, however, required the skill, hard work, and 
cooperation of hundreds of people. The project described here is the 
evolving creation of many people working together. If I am the 
conductor, then four people can be said to be playing key parts as 
"section leaaars:" Ellen Lieberman-Schreihofer, who is Assistant 
Project Director for Research and Administration; John Foster, 
Assistant Project Director for Software Development; Steve Ehrmann, 
the Annenberg/CPB Project Officer who has always been available for 
good and timely advice; and Ron Rice, who serves as Chairperson of 
the Evaluation Panel. The software development team included Murray 
Turoff, Irina Galperin, B.J. Gleason, Tod Gordon, Heidi Harting, sal 
Johar, Roland Sagolla, Sidney D'Souza, and Abdo Fathy Youssef. 
Research and administrative support was contributed by Bob Arms, 
Judith Ennis, Tanmay Kumar, 3.V. Sudarshan, Cindy Thomas, and oina 
Vora. George Baldwin volunteered his help in conducting intensive 
interviews with a small number of students. The offices of the 
Registrar and Public Relations at NJIT and Upsala were particularly 

* 

cooperative in contributing their time to the project. Faculty 
members who developed and offered online courses or portions of 



courses and who endured the extenhsive demands of the evaluation 
procedures included Lincoln Brown, Roseann Dios, B,j. Gleason, Glenn 
Halvorson, Linda Harasim, Enrico Hsu, Robert Meinke, Sylvia K, Rudy, 
and Mary Swigonskl. The full Advisory Board is listed in the 
Appendix, including identification of those who took on the arduous 
duty of se.rving on the Evaluation Panel; they have made many valuable 
suggestions which helped a great deal in setting the priorities for 
the project. Finally, the cooperation of the participating students 
is also fundamental, and I am grateful to each one who has filled out 
questionnaires, sent a bug report, or shared an idea for improvement 
in procedures. 
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EXE< i: . .viMARY 
A VIRTUA; .1/... ,^00M ON EIES 
FINAL ''AI.T.'A -ION REPORT 

The Virtual Classroom [TMj ii> a system for learning a^^d 
conununicating via connected computers. Students in the Virtual 
Classroom share their thoughts, questions and reactions with 
professors and classmates using computers equipped with specially 
designed software. The software enables students to send and receive 
messages, interact with professors and classmates, read and comment 
on lecture material, take tests and receive feedback, and more, 
without having to attend scheduled classes. Learning can take place 
at any location in the world and at any time of the day using a 
computer on campus, at home or in the workplace. 

The primary goal of the project is to demonstrate that it is 
possible to use computer-mediated communication systems to improve 
access to, and the effectiveness of, post-secondary educational 
delivery. The most im.portant "product" of the project is knowledge 
about the advantages and disadvantages of this new technology. The 
two key research questions that arise are: 

Is the Virtual Classroom a viable option for educational delivery? 
That is, are outcomes, on the whole, at least as good as outcomes 
from face-to"face, traditional classroom courses? 

What variables are associated with especially good and especially 
poor outcomes in this new teaching and learning environment? 

During the past two years, with major funding from the 

Annenberg/CPB Project, New Jersey Institute of Technology has 

constructed a prototypical Virtual Classroom, offering many courses 

fully or partially online, student^:, and professors, using personal 

computers, communicate with each other through a larger, centralized 

computer running a computer-mediated commun:.cation system called EIES 

1 . ' J 
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(Electronic Information Exchange System) , that was enhanced with 
special suftware to support educational delivery. EIES runs 
specifically on a Perkin-Elmer Corporation computer which resides at 
NJIT. However by the fall of 1988, an IBM mainframe version of the 
Virtual Classroom will be made available for lease. 

The final evaluation report summarized here includes a 
description of the software developed and of the quasi-experimental 
research design used to assess its effectiveness as compared to 
traditional classrooms. The first volume of the report focusses on 
the results for students, while the second volume presents the 
accumulated wisdom of the faculty members who took part in the 
experiment. 



SUMMARY OF VOLUME I 



Software Innovations 

Conceptually, we divided these into three types: 

. "Branch Activities" can be attached to a claws conference in order 
to support special types of assignments, or delivery of material 
for activities that involve the whole class. An "activity" is an 
executable program rather than ordinary text. For example, initial 
activity types include reading of long documents, examinations, 
t iitional q\iestion and response delivery, compiling and running 
Pascal or Fortran programs, and selection of choices from a list. 

. Support tools help the instructor manage assignments, grading and 
quizzes ^or individual students. Instructional management tools 
include an electronic gradebook and routines to collect and track 
the submission of assignments. 

. Personal TEIES [TM] is microcomputer-based software which 

integrates the composition and display of graphic elements mixed 
with text, and manages the uploading and downloading of material. 
It provides a blackboard-like facility for the Virtual Classroom. 



Col laborative Learning Strategies 

Computer-Mediated Communication is particularly suited to the 

implementation of collaborative learning strategies or approaches. 

Collaborative learning means that both teachers and learners are 
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active participants in the learning process, in this environment, 
knowledge is not something that is "deliverea" to students, but 
rather something that emerges from active dialogue among those who 
seek to understand and apply concepts and techniques. All courses in 
this project attempted to include collaborative learning elements. 

Research Methods 

In order to explore' our key rescsarch questions, we observed a 
variety of courses, students, and implementation environments. The 
primary research design is based on matching but "non-equivalent" 
sections of the same course taught in the Virtual Classroom (VC) and 
in the Traditional physical Classroom (TC) . Though the same teacher, 
text and other printed materials, and midterm and final exams were 
used, the classes were "non-equivalent" because the students were 
able to select the delivery mode. The matching courses included 
Introductory Sociology at Upsala College, freshman-level 
Computer-Assisted Statistics at Upr.ala, Introduction to Computer 
Science at NJIT, and an upper-level course in statistics at NJIT. 
The two colleges provided very different implementation environments. 
Upsala is a small liberal arts-oriented college with one 
microcomputer laboratory and little prior integration of computing 
into the curriculum. NJIT is a technological university wnere for the 
last three years incoming freshmen have been issued IBM-PC compatible 
microcomputers to take home, and where computers are used in all 
freshman- level courses. 

In the study several other courses and sections were included in 

order to increase the number of subjects and the general izability of 

the findings. Three online courses were repeated in order to allow 

the instructors to try to improve them, based on experience. Some 

other courses were taught through a combination of online and 

3 



traditional approaches (mixed mode) , one of these mixed mode courses 
was NJIT's management course for majors in other fields (OSS 471), 
which had one section that conducted its management laboratory 
exercises in the traditional manner (offline) / and one which used the 
VC as a "Virtual Laboratory." Other courses which used VC in a mixed 
or adjunct mode included Organizational Communication, a Freshman 
Writing Seminar, an Anthropology course on North American Indians, 
and a course in Business French (all at Upsala) . 

The project also included some data collection on courses 
offered online to distance education students by other institutions: 
the media studies program offered by the New School through Connected 
Education on EIES and a graduate-level course offered by the Ontario 
Institute on the PARTIcipate system. In all, data were collected 
from a total of 150 students in completely online courses. 111 in 
mixed-mode courses, and 121 in traditional or "control" courses. 

Most of the data used in the study were collected through 
pre-and post-course questionnaires. However, we also gathered 
behavioral data (including grades, when appropriate or available, and 
amount and type of online activity) and qualitative observations and 
interviews . 

Implementation Problems 

The implementation of the prototype Virtual Classroom was far 
from optimal. Problems included: 

.Insufficient recruitment of students for the experimental online 
sections. 

.Opposition from faculty members who believed that the medium would 
fail to adequately deliver college-level courses and/or that it 
would be unfair competition, causing decreased enrollments in 
their courses. 

.Failure to adequately infor-m all students enrolled in the 
experimental sections concerning the nature of the educational 
experience in which they would be involved (despite explanations 
in registration material, campus newspaper articles, flyers and 

4 
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posters) . 

.Inadequate amounts and quality of equipment for student access, 
especially at Upsala. 

.Limited capacity of the central host (EIES) , which was sometimes 
saturated, resulting in slow response or busy ssignals. 

.Unfinished software tools to support the Virtual Classroom, 

including the graphics package that had been considered vital to 
;me of the courses. 

.Resistance by some students to collaborative learning. 
.Deliberate student misbehavior. 

.Impossibility of rigid experimental control which "holds everything 
constant" except the medium of course delivery. 

These problems interacted. For instance, we had initially 
anticipated only four courses involved in the experiment. Many other 
courses were later added to the study, due in part to the low 
enrollment in the experimental sections. Each additional course had 
its own unique problems and demands, increasing the overload on the 
project's limited staff. It would have been more effective to 
implement the project over a longer time period. Though some of the 
implementation difficulties were due to the pioneer nature of this 
effort, the first implementation on any campus is likely to encounter 
similar difficulties. Thus, other colleges and universities are 
advised to start small. Select one or two courses for the initial 
efforts. The staff who gain experience can become the coaches for 
subsequent expanded programs. 
Impacts on Students 

Despite implementation problems, tne outcomes of this field 
experiment are generally positive, supporting the conclusion that the 
Virtual Classroom mode of delivery can increase access to, and the 
effectiveness of, college-level education. 

The results of statistical analysis of data relating to the 

major hypotheses concerning outcomes are listed below. Initially, 
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there was a separate hypothesis that the mixed-mode results would 

not simply represent an "average" of the virtual classroom and 

Traditional Classroom modes, but might have some un'.que advantages 

and disadvantages, in the following summary, results related to this 

speculation are included in reviewing each of the other hypotheses. 

Hypothesis Is There v. 11 be no significant differences in scores 
measuring MASTERY of material taught in the Virtual and 
Traditional Classrooms. 

Finding: No consistent differences, in one of five courses, vc 
final grades were significantly higher. 

This hypothesis was tested using a quasi-experimental design which 
compared the midterm exam scores, final exam scores, and final grades 
attained by students in matching sections of five courses, in 
Computer Science, student performance tended to be significantly 
better, on the average, as measured by grades. Though there are no 
statisticalJ.y s:'.gnif icant differences for the two freshman level 
courses^ ecology and Statistics, thtse were courses in which many 

students die D or F work in both modes, and the instructors tended to 
feel that the mode further disadvantaged young, poorly motivated 
students with marginal levels of reading, writing and quantitative 
skills. 

Hypothesis 2: VC students will perceive it to be superior to the TO 
on a number of dimensions: 

2.1 CONVENIENT ACCESS to educational experiences (supported): 

Students rated the VC as more convenient than the TC. 

2.2 Increased PARTICIPATION in a course (supported). 

2.3 Improved ability to apply the material of the course in new 

contexts and EXPRESS their own IDEAS relating to the material. 

Finding: Increased confidence in expressing ideas was most likely to 
occur in the mixed modes courses. 

2.4 Improved ACCESS to their PROFESSOR (supported). 

6 
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2.5 Increased level of INTEREST in the subject matter, which may 

carry beyond the end of the course. 

Finding: This is course-dependent. Though the averages for measures 
of increased interest are higher for both the VC and mixed 
modes, the overall scores are not significantly different. 
Interest Index scores are highest for the VC mode at NJIT and 
for the mixed -mode cotrses at Upsala. 

2.6 Improved ability to SYNTHESIZE or "see connection among diverse 

ideas and information." 

Finding: No significant differences overall; mode interacts with 
course. 

2.7 COMPUTER COMFORT: improved attitudes toward the use of computers 

and greater knowledge of the use of computers (supported) . 

2.8 Increased levels of communication and cooperation with other 

students in doing coursework (Group COLLABORAiTON) . 

Findings: Mixed and course-dependent. For example, although 47% of 
all students in VC and mixed-modes courses felt that they had 
communicated more with other students than in traditional 
courses, 33% disagreed. The extent of collaborative learning 
was highest in the mixed-mode courses. 

2.9 Improved Overall QUALITY, whereby the student assesses the 

experience as being "better" than the TC in some way, involving 
learning more on the whole or getting more out of the course 
(supported) . 

Though the average results supported most of the above 

predictions, there was a great deal of variation, particularly among 

courses. Generally, the above outcomes were dependent more on 

variations among courses than on variations among modes of delivery. 

The totally online upper level courses at NJIT, the courses offered 

to remote students, and the mixed-mode courses were most likely to be 

perceived by the the students as "better". 

Hypothesis 3: Thosf; students who experience collaborative learning 
in the Virtual Classroom are most likely to judge the outcomes of 
online courses to be superior to the outcomes of traditional 
courses . 

Finding: Supported by both correlational analysis of survey data and 
qualitative data from individual inteirviews. Those students who 
experienced high levels of communication with other students 
and/or with the professor were most likely to judge the outcomes 
of VC courses to be superior to those of TC courses. 
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Outcomes are Related to Student Characteristics In many cases, 
results of the quantitative analysis are inconclusive in determining 
which is "better," the VC mode or the TC mode. The overall answer 
is, "it depends." Reported outcomes related to Hypothesis 2 above 
are superior for well-motivated and well-prepared students who: have 
adequate access to the necessary equipment; take advantage of the 
opportunities provided for increased interaction with the professor 
and other students; and actively participate in a course. Students 
lacking the necessary basic skills and self-discipline will do better 
in a traditionally delivered course. Critical to whether or not the 
VC mode is "better" is the extent to which the instructor is able to 
build and sustain a cooperative, collaborative learning group. It 
must be noted that it takes new types of skills to teach in this new 
way. 

The VC is not without its disadvantages, and it is not the 
preferred mode for all students (let alone all faculty) . Students 
(and faculty) report that they have to spend more time on a course 
taught in this mode than they do on traditional courses. Students 
also find it more demanding, since they are asked to play an active 
part in the work of the class on a daily basis, rather than just 
passively taking notes once or twice a week. For students who want 
to do as little work as possible in a course, the Virtual Classroom 
tends to be perceived as an imposition rather than an opportunity. 

TEACHING EFFECTIVELY ONLINE: A SUMMARY OF VOLUME II 
Getting Started 

In order for students to participate effectively in the Virtual 
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Classroom, they must have adequate access to the system, feel 
comfortable with the medium and with each other, and know what is 
expected of them. To create these conditions, the instructor must be 
competent in using the system and have a course design worked out 
ahead of time, one appropriate to the medium and the capabilities of 
the specific system and students. Before trying to teach an entire 
course online, it is a good idea for an instructor to observe and 
participate in conferences conducted by others, and to practice using 
the editor and the advanced features of the software that will be 
used. It is preferable for a faculty member to begin teaching in the 
Virtual Classroom by conducting a mixed-modes (part VC and part TC) 
course. Faculty feel that, with practice, they gain a great deal of 
skill in teaching this way and that the amount of time and effort 
required decreases dramatically with experience. 
Teaching Techniques 

Responsiveness to the students is the single most important 
attribute of an effective online teaaher. This requires daily 
attention (about 30-60 minutes a day) . The instructor must act as a 
discussion leader and stimulator of active participation, and as a 
coordinator of and advisor for collaborative learning activities. The 
instructor must also establish procedures by which individuals can 
organize an«1 monitor the heavy flow of material that occurs in a 
successful VC. 
Mixed-Medi i Courses 

It is assumed that all VC-based courses are multi-media in the 
sense that text books, readings and other print-based materials are 
used by students. Lengthy materials available in print should be 
distributed that way, not put into a computer system to be read on a 
CRT. 
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However, the VC can be used to supplement courses delivered 
primarily face-to-face or via distance education modes such as audio 
and video. For example, it has been used to: 



.Serve as a "Bulletin Board" where updated information on 
assignments or exams is posted for students to check between 
v«lasses. 

.Act as "electronic office hours" for student communication with the 
instructor. 

•^!^!u^\^ medium for students to submit assignments and receive 
feedback. In some cases, this has extended to thesis advisement 
or independent study guidance. 

.conduct public tutorials. Questions and answers from students are 

°" assumption that if one student has a 
problem with a subject covered in class or in the text, other 
students may be encountering the same difficulty. 

.Facilitate group projects, providing a working environment without 
having to meet at the same time and place. 

For such adjunct use of VC to be successful, students must see 
the online segment of activity as important enough to motivate them 
to use the system frequently and participate actively. In some 
distance education courses, students have been encouraged, when 
needed, to get online and send questions to their instructor, if this 
was entirely optional and other students were not informed of, or 
responsible for, issues discussed in these exchanges, few students 
bothered to sign online at all. 

When using VC in an adjunct mode, the instructor must stress 
that it is a course requirement. It must be stated clearly that 
grades will be related to the amount and quality of students' online 
activity--" undergraduates seem to respond primarily to this motivator 
("Will it be on the test?"). Online activities should be spread 
evenly throughout the course, as oppostid to a few scattered 
assignments so far apart that students never get in the habit of 
signing on at least twice a week, and forget how to use the system 
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between sessions. Generally, a course Lhat is approximately half 
online and half via other modes is a good mix. 

Finally just as with a totally online course, use the medium 
frequently, not just for one-to-one communication between teacher and 
student, but as a tool for group collaboration and activity. This 
extends and enhances the course activities that occur through other 
media. 

CONCLUSIONS 

The Virtual Classroom is a viable delivery option for 
post-secondary education. On the average, outcomes are at least as 
good as outcomes for traditional courses, while access to educational 
opportunities is improved. The average student who participated in 
this experiment reported an improvement in both the access to, and 
the quality of, the educational experience. 

However, improved outcomes are contingent upon providing 
adequate access to equipment, faculty effort and skill in teaching 
with this new tool, and student characteristics, students who are 
motivated, self-disciplined, and possess average or better 
quantitative and verbal skills (as measured by tests such as the SAT) 
are likely to experience superior outcomes, as compared to 
traditional courses. Students who lack motivation and basic college 
level skills, or who must travel to use a computer terminal for 
access, are more likely to drop out of an online course, to 
participate more irregularly, and to perform more poorly than in a 
traditional course. 
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C'^APTER 1: INTRODUCTION AND OVERVIEW 

Perhaps a scenario is the next best thing to "being there" for 
understanding what a "Virtual Classroom" system is like. Picture a 
snowy Saturday afternoon in early December. Jenny Smith pours 
herself a mug of coffee, t>irns down the volume on "Twisted Sister" 
slightly, and decides to "go to class." She powers up her Personal 
computer, presses the key for auto-dial, and she's there. 

The first thing Jenny does is check her waiting messages. Her 
professor has graded the Fortran assignment she turned in online two 
days ago and commented on it ("A careless error in line 34, Jenny. 
Also take a look at Bob's assignment for a somewhat more elegant 
solution. Grade: 85"). Then she checks the gradebook to see what 
her average now is: 88, she's going to have to do a really solid A on 
the final exam to get an A in the course. Then Jenny joins the class 
conference. She picks out the "branch" where assignments are 
deposited. There's a special program that allows you to look at the 
other students' assignments only after yours is completed too. She 
finds Bob's program, and lists it. Hmmm. . . yes, that was a better 
way to handle that part of the problem. 

Last night, she had read the assigned textbook chapter for the 
last unit of the course, she notes the last lecture is in the class 
conference, and downloads it to her PC. Later, she will print it and 
read it carefully, using a highlighter to mark the parts she will 
want to review before the final. 

An informal "one-liner" appears on her screen: "Hi Jen— Wanna 
chat?" (Her account is set to allow others to interrupt with "real 
time" messages) . 

"Hi Sam— not unless you provide a virtual fireplace and some 
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marshmallows, " she types back. 

Jenny spends about 20 minutes reading the latest comments by 
other students in the debate about artificial intelligence, (is it 
possible? What is it? Is it good or bad?) she adds a comment of her 
own, then decides to check into the "cafe" before leaving, where 
there is a discussion going on about surrogate motherhood. That's 
not part of the course, but sort of an "extra-curricular activity," 
like going to the school pub, that students and professors from many 
courses can join. Later tonight, when she has studied the lecture, 
she will sign on again and take the weekly quiz. Jenny works full 
time, and tries to do most of her work for the course on the 
weekends. 

***** 

A "Virtual Classroom" can be defined as a teaching and learning 
environment located within a Computer-Mediated Communication System 
(CMCS). Rather than being built of bricks and boards and metal, it 
consists of a set of communication and work "spaces" and facilities 
constructed in software. In order to be considered a "Virtual 
Classroom," the system must support all or most of the types of 
communication and learning activities available in the "traditional" 
(physical) classroom and campus. There should be an interaction 
space like a classroom where the "teacher" or others may "lecture" 
and where group discussions may take place; a communication structure 
like "office hours" where student and teacher may communicate 
privately; the ability to administer, collect and grade tests or 
assignments; and the ability to divide a larger class Into smaller 
working or peer groups for collaborative assignments, ideally, there 
should also be the equivalent of a "blackboard" where diagrams or 
equations may be posted for discus.sion or note-taking. 
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One difference between the two learni:.g environments is that in 
the Traditional Classroom (TC) , most interaction takes placid by 
speaking and listening (though it may be supplemented by writing and 
reading from a blackboard or from "handouts.") In the Virtual 
Classroom (VC) , interaction takes place almost entirely by typing ar.i 
reading from a computer terminal (though it includes the use of print 
materials such as textbooks, and may be supplemented by an occasional 
face-to-face meeting or telephone call). Because it is located 
within a CMOS, interaction among teacher and students in the Virtual 
Classroom is also asynchronous, with the computer storing waiting 
communications for each participant. 

Using the analogy of software structures to emulate 
interactional forms in the traditional classroom gives the 
unfortunate impression that the VC can never be more than a 
second-best simulation of a TC. On the contrary, a collaborative 
learning environment that is computer-mediated can support some types 
of activities that are difficult or impossible to conduct in 
face-to- face environments, particularly if there is a large class. 
In addition, discussion and communication about the course becomes a 
continuous activity, rather than being limited to a short scheduled 
time once or twice a week. Whenever a student has an idea or 
question, it can be communicated, while it is "fresh." 

Both face-to-face and CMC as modes of communication have 
strengths and shortcomings (See Hiltz, 1986a). The relative 
effectiveness of a VC is conti ^nt on the teacher conducting the 
course in a manner which fits the characteristics of the medium, the 
nature of the course materials, and the characteristics of the 
students, it depends on whether or not teachers and students take 
advantage of its potential to support an active learning process that 
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incorporates extensive interaction among students and between 
instructor and students (Hiltz, 1986b). It al& • requires adequate 
access to the necessary equipment (PCs and modems), so that the 
students may easily access the facility. The basic pr emise of this 
project is that given the right software tools and depending on these 
contingencies, the VC can actuallv be a more effectiv e mode of 
delivery for post - secondary ed ucation than the TC. 

At least equally important as comparisons to face-to-face 
delivery modes would be comparisons to non-interactive forms of 
distance learning, such as the correspondence course or a television- 
based course, such comparisons were not included in this study, and 
are an important focus for future research. For instance, one might 
compare the same course delivered via television broadcast, conducted 
totally via the Virtual classroom approach, or offered in a mixed 
modes format which combined T.V. broadcasts with online discussion 
and assignment submission. 

This document describes the goals of the Virtual Classroom 
project, its implementation and use in a prototype form, the 
theoretical framework which guided the implementation, the evaluation 
methods, and the results. The primary goal of the evaluation was to 
determine the exchangeability of the outcomes of student experiences 
in the Virtual Classroom with those in the traditional classroom ; and 
to identi fy characteristics of students and of online interaction 
wtiich were associated with the most success ful outcomes for the vc 
environment . Particular emphasis was placed upon the extent to which 
educational processes in the Virtual Classroom facilitate 
collaborative or peer group learning, whereby students learn through 
communication with one another. In addition, attention was paid to 
capturing and documenting implementation problems. 
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In order to explore these questions, it was necessary to observe 
a variety of courses, students, and implementation environments. The 
primary research design rested upon matched but "non-equivalent" 
sections of the same course taught online and in the traditional 
classroom. Though the same teacher, text and other printed 
materials, and midterm and final exams were used, the classes were 
"non-equivalent" because the students were able to self-select 
delivery mode. The matched courses included Introductory Sociology 
at Upsala College (Soc 150) ? freshman-level Computer-Ass is te.. 
Statistics at Upsala (cci40y) ; Introduction to Computer Science 
(CIS213) at NJIT; and an upper-level introductory course in 
statistics for engineers at NJIT (Math 305, Statistics for 
Technology) . The latter three courses were repeated online in the 
Spring of 1987, in order to allow the instructors to improve their 
online courses, based on their experiences the first time, and to 
increase the number of subjects in the study. 

The two colleges provided very different implementation 
environments. Upsala is a small liberal arts-oriented college with 
one microcomputer laboratory and little prior integration of 
computing into the curriculum. NJIT is a technological university 
where for the last two years, incoming freshmen have been issued 
IBM- PC compatible microcomputers to take home, and computers are used 
in all freshman- level courses. 

In addition, some courses were taught with mixed modes of 
delivery (partially online and partially face-to-face) . This 
included the extensive laboratory component of NJIT's introductory 
management course (OSS 471) , which had for two semesters one section 
that conducted its management laboratory exercises in the traditional 
manner (offline), and one which used the VC as a "Virtual 
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Laboratory." Other courses which used VC in a mixed or adjunct mode 
included Organizational Communication, a Freshman Writing Seminar, an 
Anthropology course on North American Indians, and a course in 
Business French (all at Upsala) . The project also included some data 
collection on courses offered online to distance education students 
by other institutions: the media studies program offered by the New 
School through Connected Education on EIES, and a graduate-level 
course offered by the Ontario Institute on the PARTIcipate system. 

Most of the data used in the study were collected with a pre and 
post-course questionnaire. In addition, we also have more 
"objective" or behavioral data, including grades (when appropriate or 
available), and amount and type of online activity; plus qualitative 
observations and interviews. 

The sections which follow provide the background for the 
remainder of this report. They describe the project goals; summarize 
some related studies on teaching methods and the measurement of 
educational outcomes; summarize characteristics of CMC that may be 
related to its use as a mode of educational delivery; describe the 
software tools that were developed to enhance CMC for educational 
delivery; and present the theoretical framework and hypotheses that 
guided the study. 
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PROJECT AND EVALUATION GOALS 



The goal of the "Virtual Classroom" is to improve access to and 

the effectiveness of post-secondary education. 

As Ehrmann (1988, p. 2) points out, 

Access is a problem for virtually all students. The 
most severe access problems tre faced by people who, for 
reasons of location, job, handicap, economic or cultural or 
linguistic disadvantage, age, or other factors cannot 
enroll in a degree program. But access problems also 
impede students who are enrolled. Part-time or full-time 
jobs may make it difficult to attend the particular classes 
these students most need. They may have time for study, 
but not when other students are available for a study 
group. Sometimes the instructional resources they find may 
be suitable for the average learner, but not for their 
exceptionally high abilities or their unusually weak 
preparation. 

"Access" in this broad sense my be improved by the Virtual 
Classroom in the following ways: 

.Students may take any course from any instructor from any 

institution in the world which is offering courses in this mode. 
Thus, they are not limited to courses and degree programs 
offered in their geographic locality. 

.Students may participate at any time of the day or night that they 
have the time and the inclination. Opportunities for feedback 
from the instructor and interaction with other students are not 
limited to a few fixed times per week. 

.Students for whom travel is difficult may work from the relative 
comfort and convenience of their homes. This might include the 
handicapped, the aged, or those who must be at home as much as 
possible to care for children or other dependents. 

. For non-resident students, the time normally spent commuting to 
and from campus (and finding a parking space) can instead be 
devoted to coursework. 

. The technology makes it easy to exchange information that is 
difficult to share or disseminate in the traditional classroom. 
For example, a program as well as the output from a run may be 
passed back and forth among students or between student and 
instructor, for discussion of problems or bugs. They may be 
given the privilege of looking at the drafts or completed 
assignments of other students, in order to comment, compare, nr 
offer constructive criticism. CMC also allows all students an 
equal opportunity to ask questions and make comments, even if 
they have difficulty in putting their ideas into words quickly. 
They may take as long as they need to formulate their questions 



and contributions. 

However, it must also be recognized that, at least when used as 
the sole means of educational delivery, access may be limited in the 
following ways: 

.Currently, only a few institutions offer a few courses online, if 
a student wishes to complete an entire degree program online, 
the choice of courses is severely limited at present. 

.Students who do not have a microcomputer and a modem at home or at 
work will have to travel to use the necessary equipment, and 
will be disadvantaged relative to those who do have the 
equipment which makes access convenient. This is likely to be 
related to socio-economic status, since the poor are not likely 
to own microcomputers, modems, etc., or to have jobs which 
provide them with such equipment. 

However, lack of equipment need not be related to ability to 
pay. For instance, NJIT provides a microcomputer to all Freshmen and 
transfers who register, which is theirs to use for the four years 
that they are a student. Since the cost is "built into" the tuition, 
it is state-subsidized, and anyone with financial need may receive 
assistance which in effect pays for their use of the computer as an 
educational tool. 

.Lack of instantaneous feedback. In the face-to-face classroom, as 
soon as a question is asked, the answer may be received. In 
this asynchronous medium, it may be hours or as long as a day 
until an answer is received. Moreover, the teacher might be 
more likely not to answer at all, or to send a "group answer" to 
several related messages, which does not deal adequately with 
each one. 

Immediate feedback is possible with this medium, if the 
participants are online at the same time, students working together 
may arrange to be online at the same time, so that they can pass 
drafts back and forth and engage in near-instantaneous exchanges of 
remarks. Students may also work side-by-side in a laboratory 
setting, talking about and pointing to things on their screens. 
However, these are the exception. Most of the time, communication 
will be asynchronous, with answers to questions delayed. 
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.students with poor reading and writing skills may have their 

effective access lessened, since the only means of communication 
is based on writing (typing) and reading. 

.Lack of skill using a microcomputer, and software bugs or hardware 
"crashes," might severely hamper timely exchange of 
communication . 

Effectiveness is defined in terms of the extent to which a 

course achieves a set of learning goals for the learner. 

Effectiveness may be improved in the following ways: 

.Facilitation of "collaborative" or "group" learning in a 

peer-support and exchange environment, since students may "work 
together" asynchonously, they can do joint projects or 
collaborate in other ways even though their schedules make it 
difficult to work at the same time. 

.More "active" learning than in the traditional classroom. The 
computer forces responses and attention from the participants. 
They cannot just sit there passively and "tune out;" they must 
keep doing things in order to move through the materials and 
activities of the course. The active participation of each 
student may be "forced" by the software used, which may, for 
Instance, require each student to enter answers to a question or 
assignment before they can move on to another activity. 

.Facilitation of "self-pacing," that is, learning at a rate 
adjusted by the receiver rather than by the "sender." The 
student controls the pace; he or she may read as slowly or as 
quickly as is comfortable; may answer immediately or take a long 
time to think over a question or assignment before submitting a 
response. "Remedial" or "enrichment" modules or activities may 
be provided for those who are need more background or are 
capable of proceeding further than the average members of the 
class, and the "average student" may choose not to receive these 
optional materials. 

An example of self-pacing was noted during the pilot phase of 

this project. Students whose native language was not English spent 

more time online than those whose language was English. Having taken 

longer to read and re-read materials, however, their level of 

contribution and was equal to that of students fcr whom English was 

the native language. 

.The use of other computer resources (such as running a Fortran or 
Pascal program, simulations, or statistical analysis routines) 
may be "built into" the Virtual Classroom. Thus, students who 
could not afford to buy all this software themselves may have 
shared access to computer-based tools useful in their 
coursework. More importantly, as noted above, teacher and 
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learner may look at one another's input or output from software 
embedded in a CMC, for example, exhanging LOTUS spreadsheets and 
programs, or exchanging code and outputs for Pascal programs. 

.Complete notes are an automatic byproduct of the process. These 
are searchable and manipulatable in various ways. Thus, the 
student does not have to choose between active participation and 
having a record of the class, is he or she often must do in a 
face-to-face lecture/discussion . 

Evaluation of this project was both "formative" and "summative." 
As a formative evaluation, observational and questionnaire based data 
were used to obtain feedback on specific subsystems and features 
designed to support the educational process, in order to improve the 
functionality and ease of use of the final software designs. As a 
summative evaluation, the goals are to explore the following 
questions: 



1> What are the most effective teaching and learning processes in 
the Virtual Classroom (VC)? How do differences in process 
relate to differences in outcome, in online vs. traditional 
classrooms (TC)? For example, do students take a more active 
role online? Do they communicate less or more with other 
students? Included will be measures of amount and type of 
activity level by students and faculty. 



What are the advantages and disadvantages of this mode of 
delivery for attaining specific educational goals, as compared 
■o traditional classes? How do these vary with characteristics 
•se subject matter, teaching or presentational techniques, 
- characteristics, and access to and type of equipment 



3> Are the overall outcomes for VC and TC essentially exchangeable, 
or is one mode clearly superior to the other? Are the two modes 
so different that it is not possible to sey one is better than 
the other, just that they are very different? For example, when 
differences in student ability or motivation are taken into 
account, are outcomes such as exam scores essentially 
comparable? How do outcome measures for classes using single 
modes of student-teacher interaction (e.g., face-to-face or 
online) compare to "mixed modes" courses using a combination of 
delivery media? Is this related to differences in types of 
subject matter or student characteristics? 



4> Given the above findings, what implementation techniques and 
what applications are recommended for future use of this 
technology? 

21 

ERIC '-^ 



Note that the first two goals listed have to do with what would 
statistically be termed "within group" variance, as compared to 
"between group" variance. That is, we expect a wide range of 
variability in observed and self -reported outcomes for students in 
the Virtual Classroom setting. In terms of priorities, we were most 
interested in describing and/or explaining the variables which seem 
to be associated with especially good and especially poor outcomes in 
this new teaching and learning environment. 

The third goal is to identify the "aver??ge" outcomes for three 
modes of course delivery (VC, TC, and mixed) and to determine if 
there are any significant differences among them. 

This is an initial experiment with a limited number of subjects. 
Thus, we do not expect to be able to provide definitive answers to 
the above questions. The evaluation research is exploratory, aimed 
at identifying the most important variables associated with 
differences in course outcomes, particularly the interaction among 
student characteristics, teacher behavior, and mode of delivery. 
Further research with a larger number of students, with a wider range 
of courses and software variations, and with variations in the extent 
and strategy for employing the Virtual Classroom approach in courses, 
will be necessary to establish more precise estimates of "causes" and 
"effects" in this new eaucational environment. 
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LEARNING IN THE VIRTUAL CLASSROOM 

"Education is the structuring of a situation in ways that help 
students change, through learning, in intentional (and soiretimes 
unintentional) ways." (Johnson and Johnson, 1975, p. 2) The 
instructor who uses a Virtual Classroom employs computer-mediated 
communication to create and structure the learning situation. 
Students who take courses in a "Virtual Classroom" are expected to 
learn the course material in a variety of ways- Much of the learning 
of concepts and skills should occur independently, from reading texts 
or assigned articles, lis. aning to audiotapes, and/or using other 
computer tools such as Computer Assisted Learning software on a PC or 
mainframe software to run large programs. 

In the class conference, the instructor presents supplementary 
"electures" (electronic lectures) and leads a discussion. Here, the 
students must put what they have learned into their own words, 
answering questions about the material raised by the instructor and 
responding to the contributions of other students. 

Attached to the conference may also be various computer-mediated 
"activities" to be performed by students. For instance, there may be 
a quiz to take, or a computer program to write, compile, and run. 
Such activities are actually programs, rather than text, which ara 
triggered to run when the student chooses to "do" the activity. This 
concept of activities, above and beyond the exchange of text, is one 
of the key software innovations of the virtual Classroom project. 

For individual questions, the student may communicate with the 
instructor or other students by private message. For individual or 
team writing or laboratory assignments, an online notebook may be 
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used to create and edit material, with the results being shared with 
the instructor and/or other students in the class. 

The Virtual Classroom also offers some special opportunities, 
including: 

.Interaction and feedback may occur on a daily basis, rather than 
being available only during a few scheduled hours during the 

.Pen names may be used in contributing responses to questions or 
assignments. This may enable the student to share ideas and 
♦ experiences without embarrassment or revealing confidences. For 
instance, in a Sociology course, students used pen names in 
applying concepts of different types of socialization to their 
own childhood, and in applying concepts about factors related to 
interpersonal attraction to one of their own relationships. 

.Students may learn by taking the role of teacher, being 

responsible for summarizing the important points of a topic or 
"outside reading" for the benefit of the rest of the class. 

.Students may be forced to think and respond for themselves rather 
than passively listening to the instructor or other students. 
For instance, in one variety of the "response branch" activity 
designed for this project, students must independently answer a 
question before they can see the answers of the other students. 

.Putting questions and answers into a written form may aid 
comprehension for some students. It may also improve their 
writing skills. 

The specific types of learning activities online vary a great 
deal from course to course, depending on the subject matter and the 
skills and preferences of the teacher. Included in the Appendix to 
Volume 2 of this report is a narrative description of the classes 
which used the "Virtual Classroom" during the 1986-87 year. These 
were prepared by the instructors in response to a list of issues and 
topics to be covered, and explicitly include "lessons learned" about 
effective and ineffective procedures and assignments. 
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EDUCATIONAL TECHNOLOGY AJID EDUCATIONAL EFFECTIVENESS 
There is extensive literature on the effects of medium of 
communication on learning; on educational innovations in general; and 
on the instructional uses of computers in particular. in addition, 
there are many publications in the area of computer-mediated 
communication, and a few on the use of computer-mediated 
communication to support educational delivery. Eaah of these areas 
of previous research has relevance for predicting problems, 
opportunities, and effects in implementing a "Virtual Classroom." 

Communication Medium and Educational Outcomes 

Previous studies of courses delivered by television or other 

non-computer media tend to indicate "no difference" in basic 

outcomes. For instance, Schramm (1977, p. 28) states that 

Overall, there is no basis in the research for saying that 
students learn more or less from television than from classroom 
teaching. This does not mean that under some conditions of 
teaching some students do not learn more of a certain subject 
matter or skills from one medium or channel of teaching than 
from the other. But the results of the broad comparisons say 
that there is, in general, no significant difference . 

Each medium of communication has its advantages and 

disadvantages. Outcomes seem to be related more to the particular 

implementation of an educational use of a medium than to intrinsic 

characteristics of a medium, implementations which capitalize on the 

strengths of a medium, and which circumvent or adjust for its 

limitations, can be expected to be successful in terms of outcomes, 

while other implementations will be relative failures. Certainly, we 

know that some courses offered in the traditional classroom are more 

successful than others, and that this can be related to variations in 

the teaching skill and style of the instructor. Thus, it is not that 
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"media do not make a difference," but other factors may be more 

important than or interact with communication medium in affecting 

educational outcomes for students. A primary goal in studying a new 

medium of communication for educational delivery must be the 

identification of effective and ineffective ways of using it. Clark 

and Salomon (1986, p. 10) summarize this lesson on past research on 

the instructional impact of new media as follows: 

Even in the few cases where dramatic changes in achievement 
or ability were found to result from the introduction of a 
medium such as television. . . it was not the medium per se which 
caused the change but rather the curricular reform which its 
introduction enabled. 

The "curricular reforms" which the Virtual Classroom approach 
may enable are greater utilization of "active learning" and of "group 
learning. " 

The Computer and Active Learning 
Development of the computer as an aid in the educational process 
has thus far focused on Computer-Assisted Instruction (CAI) . In CAI, 
the student is communicating with a program in the computer which may 
provide a tutorial, drill-and-practice, or simulation and modelling 
exercises. At least for certain types of students and instructional 
goals, computer-assisted instruction (CAI) can be more effective than 
traditional methods alone. In their comprehensive review of CAI, 
Chambers and Sprecher (1980) conclude that it has many advantages 
when used in an "adjunct" or supplementary mode within a regular 
classroom, with class discussion following. Learners are forced to 
be actively involved in the learning process, and each may proceed at 
their own pace. Feedback tailored to each individual student 
provides the kind of reinforcement that will aid learning. However, 
when used as the sole or "primary" mode of instruction for distance 
learning, it appears to be effective only if there is also 



"significant" communication between teacher and student: "Primary 
CAI, and distance learning in general, may achieve results similar to 
those for adjunct CAI as Icng as there is sufficient human 
interaction accompanying the use of the CAI materials" (Ibid., p. 
336) . 

Bork (1981) has been prominent among those who have emphasized 
the possible use of the computer as a "responsive learning 
environment." Creating an "active learning situation" (Bork, 1985) is 
the prime consideration in computer applications to education, from 
this point of view. The "drill-and-practice" CAI approach has been a 
limiting and negative influence upon developing the educational 
potentials of the personal computer. Too often, people using 
computers "tend to transpose books and lectures, and so they miss the 
component of active learning which is so important" (Bork, 1985). 

Instructional Strategies: The Concept of Collaborative Learning 

CMC is particularly suited to the implementation oi 

collaborative learning strategies or approaches. Literally, to 

collaborate means to work together (co-labor) . Collaborative 

learning means that both teachers and learners are active 

participants in the learning process; knowledge is not something that 

is "delivered" to students in this process, but rather something that 

emerges from active dialogue among those who seek to understand and 

apply concepts and techniques. In the collaborative learning model, 

Education does not consist merely of "pouring" facts from the 
teacher to the students as though they were glasses to be filled 
with some form of intellectual orange juice. "nowledge is an 
interactive process, not an accumulation of Trivial Pursuit 
answers; education at its best develops the students' abilities 
to learn for themselves... Another way to say this is that 
collaboration results in a level of knowledge within the group 
that is greater than the sum of the knowledge of the individual 
participants. Collaborative activities lead to emergent 
knowledge, which is the result of interaction between (not 
summation of) the understandings of those who contribute to its 
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formation (Whipple, 1987, p. 5). 



Johnson and Johnson (1975) use the term "goal structure" to 

refer to the pedagogical strategy or structuring of relationships 

among students that is used in a course. <ve are reserving the term 

"goals" to refer to the desired outcomes, and in the quotations 

below, have changed their term "goal" to "strategy." 

Instruction can be defined as the process of arranging 
the learning situation in such a way that student learning 
is facilitated... Our theory of instruction states that 
successful instruction depends upon the following 
components : 

1. Specifying desired outcomes for the students and 
sett i rig a^^p^opriate instructional goals. 

2. Implementing the appropriate [strategy... 
Strategies] can be cooperative, competitive, or 
individual ist Ic . 

3. Assembling the instructional materials and 
resources needed to facilitate the desired learning. 

4. creating an instructional climate that facilitates 
the type of interaction among students and between students 
and teacher needed to achieve the instructional goals. 
(Johnson and Johnson, 1975, p. 3) . 

A [strategy] specifies the type of interdependence 
existing among students. It specifies the ways in which 
students will relate to each other and to the teacher in 
the accomplishment of instructional goals. There are three 
types of [strategies]: cooperative, competitive, and 
individualistic. . . A cooperative goal structure exists when 
students perceive that that can obtain their go.^1 if, and 
only if, the other students with whom they are linked can 
obtairi their goal ... A competitive goal structure exists 
when students perceive that they can obtain their goal if, 
and only if, the other students with whom they are linked 
fail to obtain their goal... An individualistic goal 
structure exists when the achievement of the goal by one 
student is unrelated to the achievement of the goal by 
other students... Usually there is no student interaction 
in an individualistic situation, since each student seeks 
the outcome that is best for himself regardless of whether 
or not other students achieve their goals. (Ibid, p. 7) 



Most distance learning has taken place using an individualistic 
or self -study strategy, with a totally individualistic learning 
strategy, CMC might speed up and increase feedback between the 
individual student and the teacher, but other students would not be 



involved in interactions related to the course material. A 
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competitive strategy might be implemented using CMC to help to 
provide motivation and a reference group for students, so that they 
could see how they were doing in comparison to other members of the 
class. However, computer-mediated communication is especially well 
suited to collaborative or "coop«rauive" learning strategies. This 
is the pedagogical approach which the instructors in this project 
tried to incorporate into theij: online classes, at least to some 
degree. One can also use mixed strategies; for instance, there might; 
be two or more groups, each of which collaborates internally but 
which also competes with other groups in the class. 

For example, most courses included one or more "seminar" type 
segments in which thf» students became the teachers. Individual or 
small groups of students were responsible for reading material not 
assigned to the rest of the class; preparing a written summary for 
the class of the most important ideas in the material; and leading a 
discussion on the topic or material for which they were responsible. 
Seminar format is generally restricted to small classes of very 
advanced students in the face-to-face situation, because it is too 
time consuming to have more than about 15 students doing major 
presentations. Secondly, less advanced students may feel ver/ 
embarassed and do not present material well in an oral report to 
their peers, and are even worse at trying to play the role of teacher 
in conducting a discussion. In the written mode, they can take as 
long as they need to polish their presentations, and the q^aality of 
their work and ideas is what comes through, not their public speaking 
skills. Other students can read material in a much shorter time than 
it would take to sit through oral presentations. If the material is 
poorly presented, they may hit the "break" key, whereas etiquette 
dictates that they must sit and suffer through a poor student 
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presentation in the face-to-face situation. Finally, it is easier 
for students to "play the role" of teacher in this medium, which is 
more equalitarian than face to face communication. Seminar-style 
presentations and discussions are thus an example of a collaborative 
learning activity which is often difficult in the traditional 
classroom, but which tends to work very well in the Virtual Classroom 
environment, even with fairly large classes of undergraduates. 

Collaborative or group learning has been given many labels in 
the educational literature, including "cooperative learning, 
collective learning, study circles, team learning..." (Bouton and 
Garth, 1983, p. 2), and "peer-group learning" or "syndicates" 
(Collier, 1980) . The various forms include a process of group 
conversation and activity which is guided by a faculty member who 
structures tasks and activities and offers expertise, its basic 
premise is that learning involves the "active construction" of 
knowledge by putting new ideas into words and receiving the reactions 
of others to these formulations: 

Students cannot simply assimilate knowledge as it is 
presented. To understand what is being said, students must 
make sense of it or put it all together in a way that is 
personally meaningful... it is as if one were to teach a 
child to talk by having the child listen in silence to 
others for the first two or three years of life; only at 
the end of the period would we allow the child to speak. 
In reality, the child learns in a continuous process of 
putting words together and trying them out on others, 
getting their reactions, and revising speech accordingly... 
An optimum context for learning provides learners with 
frequent opportunities to create thoughts, to share 
thoughts with others, and to hear others' reactions. This 
is not possible in the traditional clar>jroom (Bouton and 
Garth, 1983: 76-77). 

Collier (1980) summarizes many reports of an increased 
involvement of students in their courses as a result of grc ip 
learning structures, including better class attendance (reported by 
Field, 1973); greater expenditure of time on the work outside of 
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C.1 «.ss. (Collier, 196;j; Rudduck, 1978); greater satisfaction with the 
course (Beach, 1974; Goldschmid & Goldschmid, 1976) and an increased 
wish to pursue subsequent studies on the topic (Beach, 1974) . 
Collier also notefs that although most reports show "no difference" 
between courses based on small -group discussion and courses based on 
lectures and other more traditional modes of instruction (e.g., 
Cost in, 1972) , there are some documented cases in which knowledge 
gained by students was greater in the small-group setting (e.g., 
Blunt & Blizzard, 1973; Erskine & Tomkin, 1963; Clement, 1971). 
Finally, there are many reports that group learning enhances 
"higher-order" intellectual skills, such as the application of 
learned principles in fresh situations, critical thinking, and the 
synthesis of diverse materials (Clement, 1971; Costin, 1972; Rudduck, 
1978; Abercrombie, 1979). 

Studies of Teaching Innovations 

A number of other teaching innovations to encourage "active 

learning," "self-pacing," and/or "immediate feedback," involving 

either teaching techniques or technological devices, have been 

described in the literature. Many of these innovations have been 

reported as pedagogical successes, but they have not been diffused 

widely because of the demands made on faculty. For instance, Tarter 

(1982) describes his use of "group incentive techniques" which 

divided a class into study groups and based part of the students' 

grades on the daily quiz averages for the whole group. Though 

successful in terms of increasing student motivation and performance, 

the technique was abandoned after five years because it was too 

labor-intensive to prepare and grade daily exams. 

The "PSI" or Personalized System of Instruction (Keller and 

Sherman, 1974) emphasizes self-pacing, the use of written materials, 
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tutorial assistance for learning from student peers, and "mastery 
learning." (Students must score 90% or better on a test unit before 
moving on to another unit.) Malec (1982) reports that the advantages 
are that students learn more and like the method; the major 
disadvantage is that the method requires a great deal of pre-course 
preparation and a fairly elaborate administrative apparatus. Though 
Malec confirms that after nine years of PSI in a statistics course, 
he was still using the method, he laments that despite presentations, 
articles, and videotapes, he is not aware of a single other colleague 
at his institution who had adopted the method. 

There are thus many competing and complementary educational 
innovations, in order for the Virtual Classroom to be a "success," 
it must not only "work," but its use must diffuse among educational 
institutions, in the long run, diffusion of the innovation may be 
much more difficult and problematic than the technological progress 
on which it is based. 

Computer-Mediated Communication Systems 
CMCS's use a computer to facilitate communication among people 
who are dispersed in space or time. Although available since the 
early 1970's (Turoff, 1972), CMCS's were not widespread until the 
1980s, when personal computers became widespread in offices, schools, 
and homes. 

The most common form of CMCS is "electronic mail" or message 
systems, v;hich deliver discrete text communications from a sender to 
one or more recipients via computer networks. Message systems are 
one-to-one or one-to-many replacements for the written internal memo, 
the letter, or the telephone call. Conferencing systems are 
structured to support cooperative group work and group discussions. 

There is extensive literature on CMC, encompassing hundreds of 
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books and articles. (For reviews, see Rice 1980, 1984; Kerr and 
Hiltz, X982; Hiltz, 1986a; Steinfield, 1986; Culnan and Markus, 1987. 
For a general discussion of CMOS, see Hiltz and Turoff, 1978; 
Johansen, Vallee, and Spangler, 1979; Uhlig, Farber, and Bair, 1979; 
Rice 1984. Hiltz and Turoff, x985, discuss alternative structures 
for CMOS) . "Structure" can be provided by software tools or by 
explicit statement of guidelines for interaction. Among the 
objectives of such structuring devices are message routing, message 
sunmiarization, and social organization (Huber, 1982b; Hiltz and 
Turoff, 1985). Conferencing software usually provides structuring 
devices such as key words and sequential or trunk-and-branch 
numbering of discussion items, and often includes special roles or 
powers for a group leader, if there are data as well as qualitative 
communications involved, ranging from simple yes-no votes to large 
tables or files of information bearing on a decision, the computer 
can serve as a support tool by organizing, analyzing, formatting, and 
feeding back the data to the group. Finally, special structures can 
be designed for programs to be executed, such as a Fortran program to 
be compiled and executed, or a test to be administered. 

Early research on the social effects of CMC was aimed at 
generalizations about the impacts of the new medium. For example, 
Johansen, Vallee, and Spangler (1979: 180»181) summarize a number of 
studies with the statement that "computer conferencing promotes 
equality and flexibility of roles in the communication situation" by 
enhancing candor of opinions and by helping to bring about greater 
equality of participation. On the basis of early pilot studies 
comparing face-to-face and computerized conferences, Hiltz and Turoff 
(1978:124) conclude that more opinions tend to be requerced and 
offered in computerized conferences, but that there is also less 
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explicit reaction to the opinions and suggestions of others. 
However, the democracy bordering on anarchy which characterizes 
unstructured or "free discussion" CMC makes it difficult for groups 
to come to agreement on complex issues or problems (Sproull and 
Kiesler, 1986) . 

A second generation of research on CMC seeks a better 
understanding of the conditions under which the general tendencies of 
the medium are stronger, weaker, or totally absent. For example, 
current work at the New Jersey Institute of Technology focuses on the 
development and evaluation of a variety of new capabilities for CMC. 
The goal is to discover the interactions among task types, 
communications software, and individual or group attributes that will 
allow the selection of optimal system designs and implementation 
strategies to match variations in user group characteristics and 
types of tasks or applications. 

Much of the research on teleconferencing has focused on the 
question of the appropriateness of alternative communication modes 
for different functions. Media differ in "social presence:" the 
feeling that a medium is personal, warm, and sociable rather than 
impersonal, cold and unsociable (short, Williams, and Christie;- 1976; 
Rice^ 1984) . The paucity of non-verbal cues in CMCS may limit 
information that serves to improve perception of communication 
partners, to regulate social interaction, and to provide a social 
context for communication. On the other hand, participants may 
explicitly increase overt social-emotional expressions such as 
greetings (Duranti, 1986) and paralinguistic cues (Carey, 1980), in 
order to compensate for the missing communication channels. 

A controlled laboratory experiment on small group problem 
solving used Interaction Process Analysis (Bales, 1950) to compare 



the process and outcomes of computerized conferences vs. face-to-face 
discussions (Hiltz, Johnson, Aronovitoh, and Turoff, 1980 t Hiltz, 
Johnson, and Turoff, 1986). There were proportionately more of the 
task-oriented types of communication associated with decision 
quality, and proportionately less of the social-emotional types 
associated with ability to reach agreement, in the computer 
conferences. Some analysts have asserted that CMCS are unsuitable for 
social-emotional communication (e.g., Heimstra, 1982), whereas others 
have described high levels of social-emotional content which may get 
out of hand (e.g., Hiltz and Turoff, 1978; Rice and Love, 1987; 
Sproull and Keisler, 1986) . In designing the Virtual Classroom 
project, we desired to identify software structures and teacher 
behavior or approaches that would support the full range of 
communication necessary for effective education, including the 
social-emotional interaction necessary in order for .students to 
establish cooperative relationships with their instructor and peers. 
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SOFTWARE TOOLS FOR A VIRTUAL CLASSROOM 
A variety of educational institutions are using simple message 
systems (e.g., Welsch, 1982; Quinn, et. al., 1983) or existing 
conferencing systems to supplement traditional delivery modes or to 
totally conduct a course. ( An Appendix to volume 2 includes an 
annotated bibliography providing an abstract for all published case 
studies that could be located ) . Particularly notable are efforts by 
Harasim and her colleagues (Harasim, 1986, 1987; Harasim and 
Johnson, 1986; Davie, 1987) using PARTIcipate at the Ontario 
Institute; of Deutshman and Richards and their colleagues, also using 
PARTIcipate, at NYIT (e.g., Haile and Richards, 1984); of McCreary 
and her colleagues at Guelph, using COSY (McCreary and Van Duren, 
1987); and of Nipper and his colleagues, using COM in Denmark 
(Nipper, 1987). 

Electronic mail has been used in an "adjunct" mode to support 
classes delivered primarily via other media. For instance, Welsch 
(1982) reports that electronic mail led to a much more "interactive" 
class. Even grading became interactive, with the students arguing 
for better grades on specific papers and making iterative changes to 
their assignments. Quinn et. al. (1983) also documented a "higher 
proportion of student turns to teacher turns" in messages exchanged 
via computer than in the face-to-face classroom. In addition, 
content analysis showed that the length of responses by students was 
much longer in computer-mediated communication. These observations 
about changes in the balance and nature of interaction among the 
instructor and the class members were also documented in pilot 
studies of earlier online courses on EIES (Hiltz, 1986). 

Our own pilot studies were based on using the existing EIES 
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software to supplement traditional courses or to deliver non-credit 
continuing education courses. Though the results were promising 
(Hiltz, 1986b), it was evident that there were many limitations to be 
overcome, parti^ ilarly for standard college-level courses that 
required numerous assignments and examinations as part of the course 
work. Conceptually, we divided these into a set of structures called 
Branch Activities which could be attached to a class conference in 
order to support special types of assignments or delivery of material 
for activities that were to involve the whole class; a set of 
teaching support tools to help the instructor manage assignments and 
grading and quizzes for individual students; and micro-computer based 
software for the integration of graphical information with text 
information. 

Branch Activities for class Conferences 

BRANCH is the generic term used tc describe activities which are 

attached to comments in a conference. The conference comments form a 

linearly numbered "trunk;" and the "branches" attach to one of the 

main conference comments. All of the responses or activities related 

to that branch are gathered together there, instead of being 

scattered throughout a conference as many separate comments. Rather 

than automatically recieving everything that has been entered by any 

participant, as with comments, participants choose to undertake the 

activities in a branch only when they are ready do do so, and 

explicitly give a command, a record is kept of DONE branches and a 

review choice for branches helps users to keep track of which 

activities they have completed. While students may access only their 

own records of done and undone branches, the instructor can review 

the Bra.ch Activities status of any of the students. 

The Branch Activities subsystem was developed specif icixly to 
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support online classes or a "Virtual Classroom," but it may be useful 
for other applications. 

Currently there are three types of branches. The mosst 
frequently used for online classes is the "RESPONSE" branch. One or 
more questions for response by other conference members is contained 
in the main conference comment setting up a response branch. All of 
the responses are attached to this branch (comjnent) number. Most 
importantly, the author of a response branch can specify that each 
person MUST ANSWER BEFORE SEEING THE RESPONSES OF OTHERS. This is 
very important for making sure that each person can independently 
think through and enter his or her own ideas, without being 
influenced by responses made by others. Alternatively, the author of 
a response branch can allow participants to see responses of others 
before having an opportunity to add their own response. 

A READ branch allows essay or lecture type materials to be 
divided into sections. Each section has a title, and can fo^. read 
by selecting that section from the table of contents for the 
read branch, when you do a read branch, you can choose to read just 
some sections that particularly interest you, or the whole thing. 

SELECTION branch allows the members of a conference to choose 
selections from a list (such as a list of available topics 
for student assignments) and indicates who has chosen which item so 
far. Without such a mechanism, allocating selections to students 
would require either dictatorship by the instructor, or a barrage of 
message traffic. The selection branch procedure also has the 
advantage of motivating students to make their selections early, 
since whoever makes a selection first gets it. Finally, as soon as a 
valid selection is made, it is confirmed for the student, who may 
immediately begin work on the topic. 

38 

CO 



Some branches may be structured to allow the use of a PEN NAME, 
so that s-uudents may feel more free to communicate about personal 
feelings. If the conference moderator decided not to allow pen name 
responses to branches, then everything will be entered with the 
regular signature. 

Finally, Branch Activities may be s equenced . This means that 
the instructor in a class conference or others who are authorized to 
create branching activities may specify that two or more branches 
must be done in a specified order. This allows the instructor to 
control the order in which various activities or course modules are 
completed by a student. 

No r tter what type of Branch Activity one is concerned with, it 

is accessed through the same menu or interface: 
BRANCH CHOICE? 
Choose From: 

Get Branch (1) 

Display Branch (2) 

Review Branch (3) 

Do Branch (4) 

Modify/Delete Item (5) 

Author/Create Branch (6) 

Set Interaction Mode (7) 

Monitor (8) 

Create/Modify unit (9) 

The user who enters a question mark at "branch choice" receives 
the following explanation of the m3nui 
CHOICE WHAT IT DOES 

1 Gets the root comment for a branch item, header plus 

text. 

2 Displays the header for the root comment of a branch. 

3 Reviews all branch items and your status on 

completing each one. 

4 "Do"" branch will enable you to respond to a 

response branch, read a read branch, etc. 

5 Allows you to modify or delete a response or branch 

which you wrote. 

6 Allows you to create a branch IF the moderator of the 

conftrence gave you that privilege. 

7 Allows you to switch to a "batch" mode whereby all 

branch items print without pausing to ask if you 
want to see each one. 

8 Monitor or teacher privileges to manage the activities. 
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9 Allows organization and reorganization of individual 

activities into sequences. 

Conceptually, there is no end to the kinds of "Branch 
Activities" that can be added to a Virtual Classroom. The Branch 
Activity software consists of a set of programs which lead the author 
through the process of setting up the activity; a set of programs 
which lead the participants through actually doing each type of 
activity; and a common interface for accessing, tracking, and 
managing the whole set. For instance, with funding from ITT, we are 
currently adding an activity designed to handle the integration of 
input to and output from LOTUS 1-2-3 as a type of activity. 

We found that adding this new subsystem does create an 
additional level of complexity and learning time for the student (and 
faculty member!) However, in large classes with a number of 
assignments and activities, trying to do everything in a linear 
conference structure quickly results in a disorganized and 
unmanageable situation for both students and teachers. 

The only way to implement a special subsystem such as Branch 
Activities within EIESl is to use its fully interpreted high-level 
' '.nguage, INTERACT, while INTERACT is relatively easy to change and 
thus suited for a system under development, it runs slowly: Delays of 
30-60 seconds are not uncommon. The larger the subsystem gets, the 
more slowly it runs. 

In the new system being built called TEIES (Tailorabxe 

Electronic Information Exchange System) , activities will be an 

integral part of the architecture and will not operate particularly 

slowly. For this prototype implementation of Virtual Classroom 

structures, the decision was made to support only three types of 

Branch Activities, and to develop other special programs and types of 

activities as separate routines, not slowed down by the overhead of 
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the Branch Activities subsystem on EIESl. This next set of special 
tools relates to individual assignments, rather than to shared 
activities in conferences; thus it also differs in that the use of 
these tools was channeled through messages and notebooks, rather 
than through the shared class conference. 

Instructional Management Tools 
As both a systems analyst familiar with EIESl and Interact, and 
an instructor in the Virtual Classroom project, B.J. Gleason was in 
an ideal position to develop a series of instructional management 
routines (see Gleason, 1987, for a manual and full description). 
These included: 

.Makequiz, Quiz, and Grader— Makequiz allows an instructor to 
create an online quiz, which may consist of a variety of forms of 
questions (e.g., multiple choice or other "objective" questions, 
essay questions, or "short answer" responses such as the answer to 
a computation problem). Quiz allows the student to take an online 
quiz, and Grader guides the automatic grading and issuing of 
messages to students reporting their grades on the quiz. There is 
also a spreadsheet-like program, "Gradebook," which organizes and 
computes weighted averages for all grades for each student, and 
which students can consult to see their grades and average at any 
time. 

."Assignment" and "Handin" automatically organize and track all 
student responses to a single assignment in a designated page in 
the instructor's notebook. For large classes with many 
assignments, this can be very important, since otherwise the 
instructor would have to find, sort, ard transfer each of the 
individual assignments arriving as messages. 

.Pascal, Fortran, and Debug provide for compiling Pascal or Fortran 



programs in a "batch" or "background" mode on eies. This set of 
tools for courses involving programming allows the instructor to 
see the program as well as the compiled result, in order to 
improve ability to help students and to comment on the quality and 
correctness of their code. 

Personal TEIES: Integrating Graphics and Text 
The objective of Personal TEIES is to allow an instructor or 
student to compose and display, on a microcomputer, text that is 
integrated with simple graphics, including pictures and mathematical 
symbols. The graphics are composed using a subset of the Graphical 
Kernel System and are then encoded in NAPLPS, the North American 
Presentation Level Protocol Syntax, for transmission and storage in 
EIES, TEIES, or any other CMOS that accepts ASCII code. The initial 
version was implemented for the IBM PC and compatibles; we hope to 
implement future versions for the Mcintosh and other popular types of 
microcomputers . 

The graphical items created and displayed in Personal TEIES are 
meant to emulate a blackboard in the traditional classroom, with 
class members not only able to look at one another's drawings, but 
also able to "erase" and "redraw" an item. Because it is encoded in 
NAPLPS, rather than communicated as a bit-map, it can be transmitted 
over a telephone line; and, when versions for different micros are 
completed, a graphical item drawn on an IBM-PC compatible could be 
displayed by a user of another brand of micro. 

Unfortunately, Personal TEIES was much more difficult to 

implement in the IBM-PC environment than we had anticipated. A 

completely operational version was not ready until the end of March, 

1987. This version was used for a few exercises in Math 305, the 

other courses had to get along without the graphical capabilities 

42 



which we hau hoped to provide, (See Foster, 1986 and i?S7, for the 
initial and final specifications for Personal TEIES; Harting, 1986 
for the user's manual for version l.o. We did learn a lot from the 
limited trials with the initial version.) 

THEORETICAL FRAMEWORK 

This Study builds upon previous work on acceptance of 
computer-mediated communication systems and on teaching 
effectiveness, both in conceptualizing the variables which can be 
expected to affect the process and outcome of online courses, and in 
operational izing the measures of outcomes. 

Dependent Variables: Measuring the Success of the Virtual classroom 

"Acceptance" or "success" of computer systems is sometimes 

assumed to be unidimensional . For instance, if employees use an 

interactive computer system, then it may be defined by management as 

"successful." "Technicists" (see Mowshowits, 1981) or "systems 

rationalists" (see Kling 1980) may assume that if a system is 

implemented and being used, then the users must like it, and it must 

be having the intended beneficial impacts. However, many social 

analyses of computing assume that it is much more problematic whether 

or not systems have beneficial effects on users as individuals and on 

productivity enhancement for organizations. (See, for instance. 

Keen, 1981; Attewell and Rule, 1984; Strassman, 1985). 

Three components of acceptance of Computer-Mediated 

Communication Systems (CMCS) were found to be only moderately 

inter-related in a previous study of users of four systems: use, 

subjective satisfaction, and benefits. (Hiltz, Kerr, li Johnson, 

1985; Hiltz, Johnson and Turoff, 1986), The same t,hre« dimensions of 

"success" will be used in this study, it is expected that there will 
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be positive but only moderate correlations among the amount and type 
of use of the system made by a student; subjective satisfaction with 
the system itself; and outcomes in terms of the effectiveness of 
learning. Measures of the ef fecitiveness of learning or "outcomes" 
and of subjective satisfaction vrith the system are described in the 
chapter on Evaluation Methods. We have several key measures of 
amount and type of use: total hours of connect time, number of 
logins, number of conference comments composed, number of private 
messages sent, and number of different addressees to whom private 
messages were sent. 

The Independent Variables 
Among the theoretical and empirical approaches to studying the 
acceptance and diffusion of computer technology and its impacts on 
society, four major approaches were identified: Technological 
Detferminism (characteristics of the system) ; the Social- 
Psychological approach (characteristics of the users) ; the Human 
Relations school (characteristics of the groups and organizations 
within which systems are implemented) ; and the Interactionist or 
System-! Contingency perspective. This c].assif ication of four 
alternative theoretical approaches represents a selection and 
blending of perspectives presented in the work of Klmg (1980) and 
Mowshowitz (1981) on theoretical perspectives on computing and from 
Zmud (1979) and others who have looked at the effects of individual 
differences on the adoption of Mis and other technologies. 

Technological Det:erminants 

Rob Kling, in his review ol: theoretical approaches (1980), 
identifies the "systems rationalists" as those who tend to believe 
that efficiently and effectively designed computer systems will 
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produce efficient and effective user behavior. Mowshowit;! 's typology 
of theoretical approaches to the study of computing issues has a 
parallel category, the "technicist, " who "defines the success or 
failure of particular computer applications in terms of systems 
design and implementation" (Mowshowitz, 1981; 148). From this 
viewpoint, characteristics of the system or technology determine ufser 
behavior. For example, Turner (1984) showed that the form of the 
interface of the applications system used by social security claims 
representatives affected both attitudes toward the system and job 
satisfaction and performance. Applying this approach to prediction 
of success of the Virtual Classroom, the technological and rational 
economic factors which would be expected to be important in 
explaining user behavior include access to and reactions to 
particular aspects of the hardwe\re and software and the cost in time 
and money of using the new system compared to other alternatives for 
educational delivery. 

To the extent that these assumptions are correct, we would 
expect to find that reactions to the particular hardware used would 
account for a great deal of the variance in success. For instance, 
we would hypothesize that students with a microcomputer at home and a 
1200 baud modem would be most likely to fully benefit from this 
technology. In iaddition. we would expect to find high correlations 
between subjective satisfaction with the system, and amount of use 
and benefits. Wo would also expect to find few differences among 
courses; the samci technology should nave the same impacts on all 
classes and studcits. The relative power of technological 
determinants can be assessed by examining the results to see if they 
support these pre.lictic ns. 
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Individual Differences as Predictors 

The PSYCHOLOGICAL or "individual differences" approach to 
predicting human behavior when confronted with a new technology would 
emphasize characteristics of the individual ; attitudes and 
attributes, including "personality type," expectations, beliefs, 
skills, and capabilities (Zmud, 1979). Attitudes consist of an 
affective dimension involving emotions ("Computers are fun") and a 
cognitive dimension based on beliefs ("Using this system will improve 
my education.") As applied to this study, we predict that pre-use 
expectations about the specific system will be strongly correlated 
with subsequ'^.nt use of and reactions to the system. Among the 
individual attributes which we expect to affect success are ability 
(measured by SAT scores) , sex, and ethnic group or nationality. We 
do not expect age, previous use of computers, or typing skills to 
affect use or outcomes, but we included them in order to check for 
these influences. Measures of these variables are straightforward; 
the specific proposed questions may be seen in the Appendix. 

The personality-level attributes that we expect to affect 
success have to do with self-discipline, which may be related to 
perceived Sphere of Control ; we predict a moderate relationship 
between measures of Sphere of Control and acceptance. 

Spher e of control — Work on the conceptualization and 
measurement of "locus of control" built for many years on the work of 
Rotter (1966), who devised a single scale to measure Internal vs. 
External Locus of Control. Paulhus (1983; ste also Paulus and 
Christie, 1981) devised a new set of thirty items based on a theory 
of three s«3parate "Spheres of Control" (SOC) that could vary 
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independently. Personal Efficacy as a sub-scale measures control 
over the nonsocial environment, as in personal achievement being a 
result of one's effort rather than "luck." Interpersonal Control 
measures control over people in dyads and groups. Sociopolitical 
control refers to control over social and political events and 
institutions. A confirmatory factor analysis, correlations with 
measures on other scales, and experimental research which predicted 
behavior on the basis of SOC subscale scores supported the 
reliability, validity, and utility of the three subscales. 

For this study, the personal efficacy and interpersonal control 
scales are included in the baseline questionnaire, in the section 
labelled "images of yourself." The items for the two sub-scales are 
inter-mixed. 

Group or Course Differences 

The HUMAN RELATIONS approach "focuses primarily on 
organizational members as individuals working within a group setting" 
(Rice, 1984). The small groups of which an individual is part are 
seen as the most powerful determinants of behavior. From this 
perspective, participation in the decision to use the Virtual 
Classroom, user training and support, the nature of existing ties 
among group members, and the style of teaching or group management 
(electronic or otherwise) are crucial determinants of the acceptance 
and impacts of a new computer or communications technology. Based on 
this theoretical perspective, we expect large differences among the 
courses in which the students are enrolled, corresponding with 
differences in social interaction among the groups and in skill and 
level of effort of the teacher. 

Two families of theoretical perspectives are not tested in this 

study* Kling (1980) refers to them as "organizational politics" and 
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"class politics." The organizational politics approach will 
undoubtedly be fruitful in trying to understand resistance to this 
innovation in some organizations. However, it would require samplxng 
organizations and identifying Virtual Classroom proponents and 
opponents within them, rather than sampling users of the system in 
only three organizations, as we have done. It will be useful in 
assessing diffusion of the software to other organizations. The 
latter theoretical approach, which is paralleled by Mowshowitz's 
(1981) category of "radical criticism," is an ideological perspective 
that views computer technology as a new form of exploitation of the 
working class by capitalists. The impacts of computer technology are 
assumed to be harmful to society. We did not include hypotheses and 
data collection techniques which could test the relative power of 
this perspective. 

The Interaction or Systems Contingency Model 

The "Interactionist" (Markus, 1983) or "Systems Contingency" 
(Hiltz, 1986) approach to the social impacts of computing was adopted 
for this study, in this model, no single one of the above three 
classes of variables is expected to fully account for differences in 
success of the Virtual Classroom; all are expected to contribute. 
However, these sets of variables are not simply additive; they 
interact to form a complex system of determinants. For example, 
student ability and attitudes are presumed to interaci: with 
educational technology: favorable outcomes are contingent on certain 
levels of student ability and motivation. This theoretical 
perspective can be eqiaat with what Kling (1980) calls the "package" 
or interactionist approach to the social impacts of computing. In 
Mowshowitz's classification, we are termed "pragmatists, " taking the 
position that "the use made of 'omputers is determined in part by the 
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social or organizational settings in which they are introduced" 
(Mowshowitz, 1981: 150). 




EDUCATIONAL OUTCOMES TO BE MEASURED 
Educational outcomes of a delivery medium can be looked at for 
both students a;nd for faculty members. The quantitative data to be 
collected focuses upon outcomes for students. Qualitative or 
anecdotal data were relied upon to document effects on the 
instructors, since with only a handful of faculty members 
participating, statistical analysis would not be fruitful. 

Mastery 

Shavelson et. al. (1986, p. vi.) state that 

Tel'scourse evaluations must ultimately focus on 
outcomes and address the exchangeability of these outcomes 
with those attained by students in traditional courses. By 
"exchangeability" we mean the extent to which the 
knowledge, skills, and attitudes acquired by students from 
a telecourse are interchangeable with the knowledge, 
skills, and attitudes that are: (a) valued by faculty and 
administrators, and (b) acquired by students enrolled in 
the same course offered as part of the traditional 
curriculum. 

The most basic of the desirable outcomes for a course is mastery 
of the fundamental facts, concepts, and skills which the course is 
designed to teach. Such mastery is usually tested by examinations 
and assignments which are graded. Of course, a score for a ten 
minute quiz or a one-hour essay question is only a proxy measure for 
student mastery of the content of a course. Students can also be 
asked to report their impressions of the extent to which a course 
improved their mastery of concepts, skills, or facts. Post-course 
questionnaire items drawn from widely-used measures of teaching 
effectiveness were included for this purpose. We wil3 use both 
instructor-assigned grades and student self-reports to measure 
achievement of learning goals in a course. If there is no difference 
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in test scores for material presented online vs. material presented 
in traditional face-to- iace courses, we may consider this a criterion 
for minimal "success" of the virtual Classroom. 

Given that previous studies of courses delivered by television- 
or other non-computer media tend to indicate "no difference" in this 
basic outcome, (e.g., Schramm, 1977), we do not expect significant 
differences in grade distributions between VC and TC sections of a 
course. Though there may be some variation from course to course, 
depending upon the nature of the subject matter and the 
characteristics of the students, we expect that overall: 

HYPOTHESIS 1: There will be no significant differences in scores 
measuring MASTERY of material taught in the Virtual and 
Traditional Classrooms. 

Measuring Improved Writing 

Since all communication in the VC is in writing, and students 
will see one another's writing, practice in written communication may 
improve skills. Good writing in fact combines a number of skills, 
including organization, sentence structure, grammar, and the almost 
indefinable elements of "voice" and of "style" that make it 
interesting or engaging. Thus, improvements in writing skill are 
very difficult to measure. 

Computers in the form of text processors and spelling checkers 
have been used from eltmentary school on up to try to both speed up 
and improve the writing process. As Daiute (1985) points out, if 
electronic mail or computer conferencing is added to the word 
processing capabilities, one can expect some additional possible 
improvements, because after all, writing is supposed to be a "social" 
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process, a process of communication. Using the computer not only to 
assist in the manipulation of text but also to communicate it to 
others may help to provide motivation # a source of collaboration or 
constructive criticism, and a defined "audience." "Setting writing 
in a wider communication context can help students express themselves 
more naturally, even when they are writing formal essays" (Daiute, 
1985, p. 5). Moreover, "The computer conference can be a tool for 
consolidating and transmitting .'^eas in writing at a time when the 
writer feels most communicative, most excited, or most confused" 
(ibid. , p. 25) . 



As Daiute (1985, p. xiv) points out: 

With the computer as the instrument, writing is more 
like talking. Writers interact with the computer 
instrument, while the pen and the typewriter are static 
tools. The computer enhances the communication functions 
of writing not only because it interacts with the writers 
but also because it offers a channel for writers to 
communicate with one another and because it can carry out a 
variety of production activities. Writing on the computer 
means using the machine as a pencil, eraser, typewriter, 
printer, scissors, paste, copier, filing cabinet, memo pad, 
and post office. Thus, the computer is a communication 
channel as well as a writing tool. The computer is a 
1, language machine. 



Freed from the need to constantly recopy when revisions are 
made, the student using a word processing program can supposedly 
revise more easily and thus produce a better final vetssion. However, 
using the computer in the writing process can have disadvantages as 
well as advantages. (For some case studies and reviews, see 
Bridwell, Sire, and Brooke, 1986; Collins, 1982; Daiute and Taylor, 
1981; Kiefer and Smith, 1984; Malone, 1981.) Non-typists may be able 
to write much faster by hand than by using a keyboard. In addition, 
in order to write using a computer, the student has to access and 
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"power up" the equipment and software, and learn to use the commands 
of the text editing system as well as of the larger computer system 
in which it is embedded; this imposes an added burden. The few 
studies of comparative writing quality have shown that writing on the 
computer is sometimes rated lower than writing done by the same 
people with traditional tools, it may be more "floppy," because it 
is more like talking. Spoken sentences often are loosely 
constructed, and there tend to be more grammatical errors in speech, 
and more use of phrases such as "sort of" and "kind of." Computer 
drafts also tend to have more spelling errors (which may be "typos") 
and syntax errors caused by omitted and repeated words. Finally, 
"this research is not conclusive, because none of the studies have 
been dono after the writers have become as comfortable with the 
computer as they are with pen or typewriter" (Daiute, 1985, p. 113). 

The major objective of the Writing Seminar at Upsala Coll«ge is 
to improve writing. The students in one of these classes had the 
Virtual Classroom available for part of their work. All of their 
writing assignments were done in small groups online, and the 
students were asked to critique one another according to c^uidelines 
provided by the instructor. The impact on their ability to write 
clearly and well was assessed using data generated by standard 
before-and-after testing procedures at Upsala. Every Freshman is 
given a "holistically graded" written essay exam upon entrance, and 
again a semester later, after the writing course has finished. We 
took advantage of this existing data to compare changes in writing 
scores for the experimental online section with changes for students 
in the other sections. 



HYPOTHESIS 2: Writing scores will improve more for students in a 
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writing course with access to the Virtual Classroom than for 
students in similar courses who do not use the system. 

Of course, there are other factors which may affect the valicJity 
of any such conclusion. Students will not be randomly assigned to 
the various sections, and the teachers and specific topics used for 
writing assignments will vary. There is a methodological question as 
to whether this single "holistic" assessment of writing quality may 
be able to capture spooific types of improvements that may occur. 
Moreover, there is a serious question as to whether any single 
semester- long course can significantly improve writing. However, 
statistical tendencies toward a difference associated with system use 
can be interpreted as promising for morn controlled experimentation 
with writing courses in the future. 

Other Outcomes 

There are many goals related to educational process and outcomes 
that are desirable to achieve, other than high scores on 
examinations. These less tangible or higher level changes may 
actually be of more long-term value than the ability to score well on 
a test covering a specific set of subject matter material at a 
particular point in time. The capitalized words or phrases in the 
list below will be used in the remainder of this document to refer to 
the indicated outcome. The variables are given a brief conceptual 
definition below; their operational definitions are specified in 
later sections of this report. 



HYPOTHESIS 3: VC students will be more likely than TC students to 
report each of the following: 



3.1 CONVENIENT ACCESS to educational experiences. 

3.2 Increased PARTICIPATION in a course. This may be due to 

convenience or ease of participating, and may be reflected in 
the regularity and quality of their assignments, reading, and 
contributions to class discussion. Though this may be 
considered a "process" rather than an "outcome" variable, 
student participation in the activities of ci course is usually 
considered a desirable objective in and of itself. 

3.3 Improved ability to apply the material of the course in new 

contexts and EXPRESS their own independent IDEAS relating to the 
material. 

3.4 Improved ACCESS to their PROFESSOR. 

3.5 Increased level of INTEREST in the subject matter, which may 

carry beyond the end of the course. 

3.6 Improved ability to SYNTHESIZE or "see connection among diverse 

ideas and information" (Davis, Dukes, and Gamson, 1981). 
Kraworth et. al. (1964) define "synthesis" as "The putting 
together of elements and parts so as to form a whole, arranging 
and combining them in such a way as to constitute a pattern or 
structure not clearly there before." 

3.7 COMPUTER COMFORT- improved attitudes toward the use of computers 

and greater knowledge of the use of computers. This was 
measured by repeating questions on attitudes toward computers 
before and after the course, and by directly asking the students 
if they have improved their computer competence. 

3.8 Improved ability to communicate with and cooperate with other 

students in doiug classwork (Group COLLABORATION) . 

3.9 Improved Overall QUALITY, whereby the student assesses the 

experience as being "better" than the TC in some way, involving 
learning more on the whole or getting more out of the course. 



One or two items are included to measure several other possible 
desirable outcomes of a course; these were not embraced as an 
explicit objective of any of the experimental courses in this study 
and are therefore included in only a minimal way. These include 
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better "critical thinking" skills (Ennis, 1962), greater 

self -understanding, and greater understanding of ethical issues in a 

field. 

Collaborative Learning as an Intervening Variable 

Group collaboration experience has been listed above as a 
possible desirable outcome of a course. It is listed as a desirable 
objective in itself, because in "later life" people will often have 
to work together on team projects, rather than carrying out separate 
competetive efforts. "Group" or "collaborative" learning is also 
conceptualized as a key means or process in the Virtual Classroom 
environment, that may ciicl in achieving other objectives such as 
mastery of the material. For instance, whon all students are 
entering their assignments online, it is much easier to encourage 
students to look at and learn from one another's work than in the TC, 
where massive amounts of photocopying would be necessary to attain 
the same objective. However, some students may not take advantage of 
these opportunities to learn from their peers. 

• 

GROUP LEARNING was measured for all participating students with 
a set of four items included at the bottom of the "general 
information" page of the post-course questionnaire. In addition, for 
those students using the system, a number of items on the section 
labell d "comparison to traditional classrooms" were used as 
indicators. 

HYPOTHESIS 4: Those students who experience "group" or 

"collaborative" learning in the Virtual Classroom are most 
likely to judge the outcomes of online courses to be superior to 
the outcomes of traditional courses. 
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While collaborative learning experiences may also be related to 
educational outcomes in the TC, this potential relationship will not 
be explored in this report. 

There may be conflict or inconsistency among some of the goals 
and processes in the Virtual classroom. For example, self-pacing may 
conflict to some extent with collaborative learning. Irregular 
patterns of participation, though convenient for the individual 
learner, may make it difficult for groups to complete collaborative 
projects within a set time frame. In addition to examining measures 
of each of the individual processes and outcomes of interest, the 
project will assess the extent to which they are mutually supportive 
(positively correlated) , independent (not correlated) , or 
incompatible (negatively correlated) , 

Correlates of Outcomes 
In accordance with the theoretical framev/ork adopted, there are many 
factors in addition to collaborative learning experiences that are 
expected to be associated with outcomes. 

HVi^OTHESIS 5: Differences among students in academic ability (e.g., 
as measured by SAT scores or Grade Point Average) will be 
strongly associated with outcomes in the Virtual Classroom. 
High ability students will report more positive outcomes than 
low ability students. 

Good reading and writing skills are a precondition for 
collaborative learning in this environment.. An online course 
replaces all oral explanation with a writing-based discussion. 
Learning depends on asking questions and receiving responses from che 
instructor and the other students, students who lack basic 

57 rv-, 



communication skills are likely to be unable or unwilling to 
formulate questions about any dii:f iculties they are having. Since 
many of the courses included hav^i a mathematical foundation (the two 
statistics courses and the computer science course) basic ability to 
comprehend mathematical material in a written form may also be 
correlated. 



Another individual-level set of characteristics that is likely 
to be related to outcomes is attitudes and expectations. Students 
must be motivated in order to discipline themselves to sign on 
regularly and participate actively. The relevant expectations 
include attitudes toward computers, toward the system that will be 
used, and toward the course. 

HYPOTHESIS 6: Students with more positive pre-course attitudes 

towards computers in general and towards the specific system to 
be used will be more likely io participate actively online and 
to perceive greater benefits from the VC mode. 

As discussed in the section c-n theoretical perspectives, the 

personality attributes related to self-discipline and achievement 

motivation that are expected to be correlated with student behavior 

in the VC may be tapped by measureis of "sphere of control." 

HYPOTHESIS 7: Students with a greav.er "sphere of control" on both the 
personal and the interpersonaj. Ifivels will be more likely to 
regularly and actively participav:/3 online and to perceive 
greater benefits from the VC nodti. 



Students do not take courses online within a homogeneous 
context. They take a particular course, which develops a social 
structure, heavily influenced by thv. style and skill of their 
instructor in conducting the course. According to the "human 
r-^lations" approach, we would expect process and outcomes to differ 



among these groups or courses. 

HYPOTHESIS 8: There will be significant differences in process and 
outcome among courses, when mode of delivery is controlled. 
(Another way of stating this hypothesis is that there will be an 
interaction effect between mode and course) . 

Implementation issues 
Adoption of this innovation is not likely to be strongly 
influenced by findings on comparative outcomes of traditional and 
virtual classes. It is more likely to be decided on "political" and 
practical economic c rounds. 

As Shavelson et. al. note, 

The telecourse is a controversial, emotionally charged 
issue in higher education. To some it represents a threat — 
indeed, the greeiter the sophistication of the course, the 
greater the compatition and threat to traditional 
educational institutions, their curricula, and instructors. 

Case study methocs were used to document implementation issues. 

In particular, opposition to the experiment was recorded as well as 

dealt with. The practical problems of impleinenting the courses, and 

the costs in terms of time and hassles to faculty and staff, were 

described. This recording of largely qualitative aspects of t.^e 

implementation can be uised to suggest the sorts of problems and 

possible solutions which may be relevant for future implementations. 

The following is the outline of descriptive material on 

implementation which each instructor offering a completely or 

partially online course was asked to include in their case report: 

1. Description of the topics covered in the course, with a syllabus 
or outline of what was covered week-by-week. 

2. Description of the materials and activities provided for r.he 
online class (type, length, frequency). How did this differ from 
TC class materials, act.i.vities, and scheduling, and why? 




3. Description of what worked well in terms of students seeming to 
learn and to participate: and the major problems (things that did 
not go over well),. Included here might be problems with 
procrastination (uneven and delayed participation) ; software or 
hardware inadequacies; and getting students to actively ask 
questions or discuGs issues. Also included should be a section on 
any "group" or "collaborative" learning activities; how these 
worked and how they did not. 

4. This narrative case history should be produced the first time an 
online course is offered by an instructor. Later, if the 
instructor repeats an online section, a postscript should be added 
describing how the pedagogical goals or strategies were changed 
for the repeat offering, and how these changes seemed to work. 

Implementation issues will therefore be treated in a mostly 
qualitative manner. The course "case reports" by the instructors are 
included as an Appendix to the second volume of this study, and will 
be drawn upon in order to help illustrate and explain the data 
presented in this volume. 

There are two aspects of impleme.itation that can be explored 

with our quasi-experimental design and examined using quantitative 

rather than purely qualitative data. These are the effect of course 

repetition and the effect of the nature of the educational 

environment, as it varies among colleges. Some of the online courses 

were repeated a second time. Because the VC is a new approach to 

teaching, we expected that instructors would learn from their first 

attempts and improve their skills for teaching online with practice. 

Hypothesis 9: Outcomes for the second offering of a VC course by an 

instructor will be significantly better than those for the first 
attempt at teaching online. 

In addition, the Virtual Classroom v/as implemented within two 
very different educational environments. It will not be possible to 
disentangle ^hich differences between Upsala and NJIT may be most 
important in explaining any differences in outcomes. Howevtsr, it can 
be expected that these outcomes will be influenced by differences in 
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access to equipment, skill level and computer experience of the 
students, and the general "educational environment" within which the 
experiment took place. 

Hypothesis 10: There will be significant differences between the 
Upsala and NJIT implementations of the Virtual Classroom, in 
terms of both process and outcomes of the online courses. 

Two Modes or Three? 
In the hypotheses above, mode of delivery is dichotomized: 
courses using VC vs. courses conducted totally in a Traditional 
Classroom environment. The initial design for this field study 
anticipated only two modes of delivery, in fact, as actually 
implemented, we had three modes of delivery: totally VC, totally TC, 
and mixed. Is the mixed mode simply a variant of the VC, some sort 
of average of the other two modes? We have no prior studies to serve 
as a basis for answering this question, but we suspect that it is 
not. 

Hypothesis li: Results for the "mixed" mode will not represent a 

simple "average" of results for totally VC and totally TC modes, 
but will represent a distinctive set of strengths and 
weaknesses. 

Thii5 is an admittedly vague statement. What it means is that in 
each of the preceding hypotheses, we will be aware that there may be 
significant differences between VC coursos offered totally online and 
those offered in a mixed mode. 



SUMMARY OF CHAPTER 1 
The primary goal of the. project, "Tools for the Enhancement and 
Evaluation of a Virtual Classroom," is to demonstrate that it is 
possible to use computer-mediated communication systems to improve 
access to and the effectiveness of post-secondary educi?.t:ional 
delivery. The most important "product" of the project is knowledge 
about the advantages and disadvantages of this new technology, as 
they may be influenced by variations in student characteristics and 
implementation techniques and settings. The two key questions are: 

.Is the Virtual Classroom a viable option for educational delivery? 
That is, are outcomes, on the whole, at least as good as outcomes 
for traditional face-to-face courses? 

.What variables are associated with especially good and especially 
poor outcomes in this new teaching and learning environment? 

Previous studies of teaching effectiveness, acceptance of 

computer-mediatec" communication, and results of pilot projects 

employing the Virtual Classroom approach influenced the selection of 

variables and measures. This chapter has presented 11 hypotheses 

that were used to guide the data collection and analysis strategies. 
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i:ha.pt;eh 2 
research methods 

The co-existence of several evaluation goals, and the practical 
fact that the Virtual Classroom is still a relatively rare 
occurrence, led to the adoption of a dualistic evaluation plan. 
Steve Ehrmann (1986), the Annenberg/CPB staff officer working with 
the project, speaks of "uniform impacts" and "unique uses" 
evaluation. In regard to the former, one is seeking the "average" 
impacts of the new educational practice or program, and a form of 
experimental design is most appropriate, one asks what the 
educational innovation "does" to the students. The "uniform impacts" 
approach is focussed on finding out if particular types of changes 
occur at a statistically significant level, no matter how much or how 
little the "absolute" amount of such changes may be. An alternative 
approach is to ask what the teachers and the students do with the 
technological innovation. 

In the "unique uses" perspective, an educational innovation can 
be viewed as a set of incentives and resources being offered to 
students; students are the actors, not the objects. The 
"consequences" of a program are "caused" by the choices and 
characteristics of the individual instructor and the irdividual 
students within the setting. An "excellent" innovation "stimulates 
students into a range of important kinds of learning and other 
beneficial outcomes" and/or "stimulates faculty to continued 
engagement with and improvement of teaching" (Ehrmann, 1986, p. 7). 
The nature of these outcomes may differ qualitatively as ^ell as 
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quantitatively from student to rjtudent or course to course. One wants 
to know if there are any major changes: What are the most important 
things that happened? Generally "unique uses" cannot be predicted 
ahead of time. 

In evaluating, it is desirable to capture and describe cases of 
"uniqiui uses" with such "excellent" results, or, by contrast, cases 
with notably poor results. These "cases" may consist of entire 
courses, related to characteristic? of the subject matter or of the 
mode of use of the VC technology by the instructor; or, the "cases" 
inay consist of individual students, in relation to their motivation 
and ability or other characteristics. 

TARGET COURSES AND SUBJECTS 

Annenberg/CPB was interested specifically in two undergraduate 

courses, Introductory Sociology and Introductory Statistics, and was 

willing to support an Introductory Computer Science course online. 

Introduction to Sociology (SOC 150) was offered through Upsala; it is 

taken primarily by freshmen and has no prerequisites. Introduction 

to Computer Science (CIS 213) is a second-level course at NJIT, with 

a course in Fortran as the prerequisite. The -'^a* 'sties course was 

offered in two versions: a freshman-level course at Upsala with no 

mathematical prerequisites except acceptable scores on a Math Basic 

Skills test; and an NJIT upper-level first course in statistics for 

engineers, with a calculus pre-requisite. The Upsala course is 

actually a half-course; during the first six weeks of the semester, 

the freshmen take Introduction to Computers. The half -course in 

statistics is a new part of a required core curriculum. 

For these target courses, a quasi-experimental design of 

matching face-to-face and online sections of the same course, all 

offered during the fall of 1986, was sel I. The design is 
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quasi-experimental rather than a fully controlled experiment for two 
major reasons. Students self-selected mode of delivery and the 
nature of assignments differed between matched sections. Efforts 
were made to encourage students to register in the experimental 
section, but only with full understanding of its experimental nature 
as an "unproven" method of delivery. This set of courses provided 
the primary data to be used in the assessment of exchangeability of 
outcomes of the virtual and traditional classroom means of delivery. 

Initially, it had been intended to use exactly the same 
assignments in the matched online and Virtual Classroom sections of 
courses. However, the faculty members pointed out that this would be 
totally inappropriate, and would fail to take advantage of the unique 
opportunity offered by the VC for collaborative activities. So, the 
faculty members were freed to devise whatever assignments they 
thought most appropriate for this medium, provided the text books and 
the midterm and final exams were the same. 

Each instructor incorporated collaborative activities in the 
online section which were different from the individual assignments 
given in the traditional section. This varied widely depending on 
the nature of the course. For example, in the upper-level statistics 
course, students could see one another's homework assignments after 
they had done their own, in order to compare approaches, in some 
assignments, each student chose one problem to work on instead of 
doing them all; the rest of the class could see their solution. In 
Introductory Sociology, many assignments made use of pen names and 
required students to enter analyses of how general concepts, such as 
role conflict, applied to their own lives. The use of pen names 
prevented embarassment in using examples from their own experiences 
to share with the class. In Computer Science, the VC section had a 

65 

9^- K' 



final assignment requiring a group to complete a complex program by 
breaking it into subroutines, and then making sure that all the 
subroutines worked together to produce the correct overall result. 
Such an assignment was possible only for a group able to work 
together constantly, and to have an integrated facility online for 
showing programs to one another, compiling, and executing them. The 
traditional section had only simple, individual programming 
assignments. 

However, these introductory courses are not at all 
representative of the range of applications of the Virtual classroom, 
or for exploring variations in process and outcome in such an 
environment. For these purposes, the sample was expanded to include 
many other courses which used the VC mode of delivery. For example, 
whereas all the instructors had extensive experience delivering 
courses in the traditional mode, this was a "first time" experience 
teaching an entire course in a virtual Classroom. On the basis of 
this experience, they might change their minds about effective 
procedures in this new mode. It was possible to schedule online 
sections of the computer science and the two statistics courses to 
repeat in the spring semester; but not possible, given teaching load 
and limits, to also schedule a second "control" course in the spring 
of 1987. Therefore, the sample was first expanded to include a repeat 
of three courses online. 

Secondly, there are many potential applications of the "VC" in a 
"mixed-modes" format. Some part of the course is conducted 
face-to-face, and a part occurs online. A total of five courses 
using this mixed mode of delivery were included: an introductory 
management course, a writing course, organizational communication, 
anthropology, and business French. 
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The introductory management course (OSS .i71) offered at NJIT is 
a particularly interesting "mixed modes" application. This course 
aims to give seniors with majors in disciplines ouher than 
Organizational and Social Sciences sufficient knowledge and skills to 
learn "how to manage" in a single course, since many of them will 
eventually assume managerial positions within their professions. It 
had not been planned as part of the quasi-experimental study, its 
instructor, Enrico Hsu, had been a student in one of the partially 
online graduate courses conducted during the first year of this 
project. He was beginning his first year of full time teaching at 
NJIT. Two weeks before the start of the fall semester, he approached 
the project director with a plan for an online "Management 
Laboratory." It sounded like a promising and very innovative use of 
the tecl nology, there was a second section taught by the same 
instructor which could serve as a control, and so we said, "OK," not 
quite knowing what to expect, what would turn out to be one of the 
most successful applications of VC was thus an unplanned, last-minute 
addition to the project, created by an instructor who was inspired to 
design a new type of use for the technology. 

In both the fall and the spring, there was an "experimental" and 
a "control" section of this management course. The control or 
traditional section completed all course activities in the 
traditional manner. The major course assignment involved the 
organization and simulated operation of a company over a "fiscal 
year." The control sections did this by meeting face-to-face during 
one of the scheduled class times periodically, and by communicating 
by telephone or written memo or out-of-class meetings in between. 
The experimental sections carried out their management laboratory 
assignment completely online. There was a class conference "or 
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general discussion and separate conferences and notebooks where the 
simulated organizations conducted their business. In looking at some 
of the data on this course, we found that the amount of usage was 
actually heavier than in several of the courses that were totally 
online. For many analyses, therefore, this course will be included 
along with totally online courses. The Spring face-to-face section 
was selected as the "control," since the fall face-to-face section 
was inadvertently omitted from distribution of baseline 
questionnaires, and only about half of its students completed the 
post-course questionnaire. 

The applications of the mixed mode are described for most of the 
other courses in an Appendix to the second volume of this report. 
Unfortunately, the instructor for the Business French course, Dr. 
Glenn Halvorson, died suddenly just after the academic year ended and 
was never able to complete his course report. In that course, the 
conference was used for a role playing exercise throughout the 
semester, with the students writing "business letters" in French to 
one another in the conference, relating to the hypothetical 
negotiations which might be undertaken by Americans conducting 
business in France. Professor Halvorson was inspired to try this 
simulation partially as a result of hearing about the Management Lab 
application, and in fact. Prof. Hsu occasionally "dropped into" the 
scenario and took part. 

The Freshman Writing Seminar is also of particular interest. In 
addition to a class confers '•ce for general announcements and 
discussion, the class was divided into three writing groups. In 
each group, each student entered drafts of assignments using a pen 
name. They were then guided and encouraged to make constructive 
suggestions for improving one another's drafts, with these critiques 
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also entered with pen names. 

Besides the specific courses in Sociology and Statistics 
required by the terms of the contract from Annenberg/CPB, the other 
courses were included on the basis of the teaching abilities and 
interests of specific faculty members in participating in the 
experiment. The project director wished to have a variety of courses 
represented, and actively recruited faculty members who were known to 
her as good and innovative teachers, and who had used EIES in the 
past and seemed to enjoy it. 

Faculty who offered completely online courses were given two 
months during the preceding summer to prepare materials for the 
online mode of delivery; and one "released course" during the fall to 
support their additional work in offering the course the first time, 
and preparing reports for the project. No additional released time 
was given for an online course repeated a second time. Those faculty 
members who offered partially online courses were paid for five days 
total time for their preparation of reports and participation in the 
research and planning related to the project. The actual time that 
they invested in the project was generally much more than the five 
days that they were paid for; obviously, they were "believers" in the 
medium, rathfci* than a random sample of faculty members. 

There are many ongoing sets of courses which are currently being 
offered by other institutions online, but for which there is nv. 
traditional equivc lent. These include graduate level courses in 
media studies, offered through Connected Education on EIES, with 
registration and credit at the New School. Begun in 1986-87, a 
series of two-month long master's level courses is offered throughout 
the year. At least one student has already completed an entire 
master's degree online. Each student was included in the study only 
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once, even though they might have taken six or more courses during 
the year. The response rate for the mailed questionnaires to this 
group was much lower than the response rate for questionnaires 
administered or collected during the face-to-face meetings on the 
first and last days of the NJIT and Upsala courses that were totally 
online. Thus, the total number of subjects for Connect-Ed (29) does 
not reflect the total size of their student body. 

Connected education is interesting because of the extreme 
geographic dispersion of the participants. For instance, one course 
was co-taught by instructors from Tokyo, Washington D.C., and New 
York, and had students from North and South America and Asia. 
Connect-Ed has used the ability to define group commands on EXES to 
construct an entire electronic campus to support its master's degree 
program. For instance, there is a "cafe" where students and teachers 
from all courses may mingle and chat, a "library" and a periodic 
campus "newspaper." 

The "School of strategic and Management Studies" is offered 
online on EIES by the Western Behavioral Sciences Institute. A 
post-graduate series of month-long seminars for executives offered by 
internationally prominent experts and costing $25,000 for two years, 
it is another example of the unique kinds of offerings that may occur 
through this medium in the future. With no grading and a mainly 
discussion oriented process, the instruments used for undergraduates 
in this study are hardly appropriate, but WBSI did make all of the 
transcripts of its courses available for analysis, and some of its 
students completed a special short questionnaire which was used in 
compiling the qi'ide for teaching online. 

Finally, ^ ost-graduate course offered for teachers by the 
Ontario Institute for Studies in Education on their PARTI system 
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serves as an exaiaple of continuing profi»ssional education online. 
The results for this course will occasionally be displayed and 
included in the analyses. 

The purpose of including these additional courses in the study 
was to increase the overall sample size, and thus the chances of 
obtaining statistically significant results. The expanded sample of 
courses also increases the generalizability of the findings to a 
wider range of online offerings, and facilitates exploration of 
variations among online courses. 

Table 2-1 shows a categorization of the courses include and the 
number of subjects in each category „ The difference between the 
number originally enrolled and the number for which we have complete 
data is due to a combination of drop-outs and failure to complete a 
post-course qu>3stionnair«.. A few of the "missing" qxiestionnaires 
were completed, but were turned in anonymously, so that they can 
generally be used only in looking at univariate distributions. The 
total number of students in all courses in the study is 150 totally 
online, ill in mixed cnline and traditional classroom sections, and 
121 in "control" or offline sections. 

There is an unfortunate confounding in the design; both of the 
totally online courses at the Freshman level were offered at Upsala, 
and the two totally online courses at NJIT were at a higher level. 
With only four totally online courses supported by the project, 
however, it is inevitable that not all relevant variables could be 
adequately controlled. 

Research Design 

The star.dard experimental design of random assignment to matclied 
sections of traditional and experimental courses is neither 
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practical, ethical, nor particularly relevant. Students cannot be 
randomly assigned to sections of a course meeting at different times, 
given the constraints of their other obligations, and the same 
instructor obviously cannot teach two sections of the same couise at 
the same time. It is not ethical, because this is an experiment; 
there is some risk that the outcomes will not be favorable, and 
students should voluntarily agree to assume the risk of using an 
experimental form of delivery for an entire course. Finally, it is 
not methodologically sound in terms of estimating future impacts. 
Students who choose telecourses, especially teleoourses delivered via 
computer, are likely to differ f>-om students choosing traditional 
couri>es in non-random ways. They are more likely to have 
out-of-class obligations which make it difficult for them to attend 
regularly scheduled classes, for injstance, and to have more positive 
attitudes toward computers. Random assignment is also not 
ii;ethodologically sound when one of the objectives is to explore 
variations among online classes. There are many online courses for 
which there simply are no "face-to-face" equivalents, because they 
are designed specifically for distance education; and many 
traditional classes requiring laboratory equipment, such as biology 
or chemistry, for which there is no online equivalent possible at the 
present time. 

Shavelson et. al. (1986) state that three designs can be 
identified as relevant to evaluating student outcomes from 
telecourses. These are: 

1. "Uncontrolled Assignment to form Non- Equivalent Groups," in 

which students self-select into tele- or traditional 
courses. Before and ;ifter knowledge and skills are 
measured. This is the primary evaluation design chosen for 
this study. 

2. "Patched-up Design" is "appropriat when ins^itutions 
regularly cycle students through ti\e same course, such that 
students from one cycle can serve a.r a control group for 
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students from another cycle." Unfortunately, this is not 
the case at NJIT or Upsala, and the design can be used only 
to a very limi ed extent. 
3. "Case Study Methods" provide narrative (descriptive and 
cpaalitatlve) accounts. Elements of the case study method 
will be included. 

The above set of alternative methods, however, ignores the 
important question of variation in success within telecourses. In 
examining the question of "assossing interactive modes of 
instruction," Davis, Dukes, and Gamson (1981) reach the following 
conclusion: 

Low priority should be given to conventional 
evaluation studies that compare a control group using a 
conventional classroom with aa experimental group using 
some interactive technique... We doubt that fruitful, 
context-free generalizations can be found demonstrating 
that one technique is unifor-iy better than another, even 
for specific learning objt tves. 

Our alternative approach accepts the fact that these 
techniques show no evidence of general inferiority to 
conventional techniques... The focus should be on the 
conditions under which given interactive techniques are 
most and least appropriate. We need to know the contextual 
variables that maximize the effectiveness of a given method 
(321-322) . 

Given that the Virtual classroom <3 a new educational 
technology, we do not agree that it is unnecessary to prove that it 
is just as good as a traditional classroom for MASTERY of facts and 
information. For this purpose, we will follow the traditional 
evaluation approach of experimental and quasi-experimental design. 
For each of five target undergraduate courses, we are attempting to 
match the same course with the same teacher, texts, and tests in 
Traditional Classroom mode with a mode employing the Virtual 
Classroom. Examination scores and other outcomes can then be 
compared for the two sections, in other words, at the core of the 
evaluation design is a 2 x 5 factorial design, with each of five 
courses offered in two modes of '.delivery (Sen the top of Table 2-2). 

However, this basic design wi.U be supplemented with data from 
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other courses which used the Virtual Classroom in a variety of ways 



(1) The online courses which are repeated fall and spring can 
also be analyzed as a quasi-experimental factorial design 
with a 4 (course) by 2 (first vs. second offering) design 
(middle display of Table 2-2) . 

(2) We can look at differences among modes in terms of totally 
online courses vs. traditional classroom courses, vs. mixed 
mode courses; in other wordc, a one- factor, three levels of 
treatment design. This gives us the largest number of 
subjects; the number for whom at least some data are 
available is shown at the bottom of the diagram for "design 
3." 

(3) We can examine contextual factors related to the 
conditions under which VC was most and least effective. 
These include differences among courses and organizational 
settings, and differences related to student 
characteristics, attitudes, and behavior. One of the major 
contextual variables considered will be the institution 
within which a course is conducted. The third display in 
Table 2-2 shows the basic 3 (modes) by 4 (colleges) design 
for this analysis. 
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Table 2-1 
Number of students, by Course 



Course 



AT NJIT 



Period 



Mode Enrolled 



Completed 
Post- 
Course Q 



CIS 213 
CIS 213 
CIS 213 

Math 305 
Math 305 
Math 305 

Management (OSS471) 
Management (OSS471) 
Management (OSS471) 
Management (OSS471) 

AT UPSALA 

Intro Soc 
Intro Soc 

Statistics 
Statistics 
Statistics 

Organizational 
Communication 
Anthropology 
Writing Seminar 
Business French 

OTHER 

Connected Education 
Ontario Institute 



Fall Online 
Fall Offline 
Spring online 

Fall Online 
Fall Offline 
Spring Online 

Fall Mixed 
Fall Offline 
Spring Mixed 
Spring Offline 



Fall Online 
Fall Offline 

Fall online 
Fall Offline 
Spring Online 

Fall Mixed 



Fall 
Fall 
Spring 



Mixed 
Mixed 
Mixed 



All Online 
Year 
Spiring online 



17 
20 
21 

13 
22 
27 

28 
21 
32 
26 



17 
19 

14 
20 
12 

12 

12 
18 
8 



43 
12 



9 
12 
10 

9 
19 
23 

23 
13 
23 
20 



11 
18 

12 
17 
9 



8 
12 
6 



11 
7 
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Table 2-2 



QUASI-EXPERIMENTAL DESIGNS FOR ASSESSING 
DIFFERENCES IN OUTCOME BY MODE 

Number of Students for Whom 
Data are Available Shown in Cells 



Design 1 
COURSE BY MODE 



COURSE 


ONLINE 


FTF 


CIS 213 


13 


13 


MATH 305 


12 


22 


MAiTAGEMENT 


28 


24 


INTRODUCTORY SOC 


16 


19 


STATISTICS 


11 


15 


TOTAL 


80 


98 




Design 2 




REPETITION 


OF ONLINE 


COURSES 


COURSE 


FALL 


SPRING 


CIS 213 


13 


19 


MATH 305 


12 


24 


MANAGEMENT 


28 


30 


STATISTICS 


11 


11 


TOTAL 


64 


84 



Design 3 
SCHOOL BY MODE 



ONLINE MIXED FTF TOTAL 

UPSALA 41 38 26 105 

NJIT 71 58 63 192 

CONNECT-ED 13 13 

OISE 7 7 

TOTAL 132 96 89 315 
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EVALUATION INSTRUMENTS AND PROCEDURES 
Data collection and analysis is being conducted under 
"protection of human subjects" guidelines, whereby all participating 
students are informed of the goals and procedures followed in tha 
project and confidentiality of the data is protected. A variety of 
methods is being used for data collection, including questionnaires 
for students, automatic monitoring of online activity, participant 
observation in the online conferences, use of available data such as 
grade distributions or test scores for participating students, 
descriptive case reports by the instructor for each course, and a 
small number of personal interviews. 

Questionnaires 

Pre-and post-course questionnaires completed by students are the 
most important data source. (See Appendix) . The pre-course 
questionnaire measures student characteristics and expectations. The 
post-course questionnaire focuses on detailed evaluations of the 
effectiveness of the online course or course segments, and on student 
perceptions of the ways in which the Virtual Classroom is better or 
worse than the Traditional Classroom. 

The pre-course questionnaire was adnii/iistered and collected at 
the beginning of the first "training" session in which the EIES use 
comprised or supplemented the instructional delivery mode. For 
Connected Education students and OISE students, the pre-course 
questionnaire was included with the mailed system iocumentation, with 
immediate return requested. 

Post-course questionnaires were mailed to online students one 
week prior to the final examination. They were asked to bring the 
completed questionnaires to the fina} exam. The instructor collected 
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each questionnaire as the final exam was handed to each student, if 
the questionnaire was not completed, the instructor handed a new one 
to the student and asked her/him to complete it after finishing the 
exam, students were told that they could stay extra time if 
necessary to complete the questionnaire. If a student refused to 
complete a questionnaire, this was his or her right under the 
protection of human subjects regulations, and did not affect the 
course grade in any way. 

For courses in "mixed" mode, the post-course questionnaire was 
distributed and collected in class, towards the end of the semester. 
A mailing with two follow-up requests was used for Connected 
Education students and for students who were absent during an 
in-class administration and session. 
Measuring Course Effectiveness 

The items used to measure students' subjective assessments of 
courses were included in the post-course questionnaire. They were 
developed on the basis of a review of the literature on teaching 
effectiveness, particularly centra's (1982) summary. Copies of the 
available student rating instruments described in that book were 
obtained, and permission to use items from these standard 
questionnaires was requested. Effectiveness was conceptualized as 
being related to four dimensions: course content, characteristics of 
the teaching, course outcomes, and comparisons of process in the 
virtual and online formats. These dimensions are presented as 
separate sections in the post-course questionnaire, with the hope 
that the responding students might consider each dimension separately 
in their ratings. 

Not all institutions were willing to give permission to use 
items from their teaching effectiveness instruments. Among those 
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from whom permission to use items for measuring effectiveness were 

obtained and from which items were used are: 

.Center for Research on Teaching and Learning, University of 
Michigan (Many items borrowed from their "catalog" of questions 
available for instruc . -ir- designed questionnaires) . 

.Evaluation and Examination Service, University of Iowa, Student 
Perceptions of Teaching (SPOT) test item pool (many items used 
or adapted) . 

.Endeavor Instructional Rating System, Evanston ill. (a few items 
adapted) . 

.Instructor and Course Evaluation (ICE), Southern Illinois 
University at Carbondale (a few items adapted) . 

Almost all of these items from standard teaching effectiveness 
questionnaires suffer from the potential methodological problem of 
response bias. Likert-type items are worded positively, and the 
semantic differential type items are arranged so that the most 
positive response constantly occurs on the same side of the page. 
Though rewording for approximately half of the items was considered, 
it was decided to leave them in their original forms so that the 
reiiults might be more directly comparable to those for other studies 
u^ing the same items. 

Course evaluations by students are admittedly a controversial 
means of measuring course outcomes. They have been observed to vary 
with many things in addition to teacher competence and student 
learning, such as an interaction between faculty status and class 
size (Hamilton, 1980) . Student evaluations are strongly related to 
grades received in the course. There is argument about which is the 
cause and which is the effect. If grades are "objective" 
measurements of amount of learning, then we would expect that 
students with high^ir grades in a course would also subjectively 
report more positive outcomes. However, it may be that a student who 
has a good grade in a course rates that course and instructor 



positively as a kind of "halo effect" of being pleased with the 
course because of receiving a good grade, if the latter explanation 
were true, we would expect to see that student ratings on various 
dimensions are somewhat homogeneous and do not discriminate well 
among items measuring different aspects of the process or outcome 
(e.g., students with a D or F would rate everything about the course 
as poor, while students with an A would rate everything about a 
course as excellent.) Such distortions of teaching evaluations are 
probably more prevalent when the student raters know that their 
responses are being used as input for evaluating faculty in personnel 
decisions, in this caice, the participants knew that their ratings 
were used only for this research project, and the ratings were made 
before final grades were received. Despite the limitations of 
subjective ratings, the students were probably in a better position 
than anyone else to report on the extent to which they had or had not 
experie.iced various positive or negative outcomes from a course. 

Survey o.f Dropouts 

All students who dropped an online course or who requested 
transfer to the traditional sections were surveyed with a special 
questionnaire designed for this purpose. The questionnaire probed 
the reasons for the action by the student and whether they 
constituted a "rejection" of the technology or other factors (see 
Appendix) . Among these reasons might be dissatisfaction with the 
software or with response time? inadequate access to equipment; or 
reasons not r«ilat©d to the itsode of delivery, such as personal 
problems ^ dir^like for the subject matter in the course, or the work 
load required. 
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We had initially planned to have "dropouts" interviewed 
personally, either when the studen^; saw an instructor about dropping 
a course, or shortly after. However, this proved not to be 
practical. Though official regulations say that students who are 
going to drop a course should see the instructor and/or that the 
registrar should inform an instructor promptly of drops, this in fact 
does not happen, students "disappear" without forroally dropping 
until the deadline for withdrawal, right before the end of the 
semester. They apparently also forge instructors' signatures on 
course withdrawal forms. In sum, our information on course 
withdrawals has proven to be so delayed that an immediate personal 
interview could not be conducted. 

Dropouts who did not respond to the mailed questionnaire (with 
two mailed follow-ups) were contacted several times in order to try 
to interview them by telephone. They turned out to be very hard to 
reach; the Appendix includes the one telephone interview which we 
were able to obtain. 

Automatic Monitoring of Use 
We are using and refining software built into the current EIES 
system for measuring the amount and type of online activity by 
participants. A routine on EIES called cONFerence ANalysis (CONFAN) 
permits the tabulation and display of the number and percentage of 
lines and items contributed by each member of a conference, either 
for a specified part of the conference or for the entire conference. 
This automated analysis was run for each class conference. We will 
need to extend this capability in the future so that measures of 
participation in the "branches" can also be gathered and displayed. 
For this study branch responses were manually counted and included in 
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the results of the CONFANS. 

Monthly "billing group" data available for each member of a 
billing group during the previous calendar month were recorded for 
the following: 

.Total number of conference comments contributed. This is not a 
complete measure of student activity related to the class, since 
it excludes contributions made in "branches" (which were numerous 
for some courses), or in notebooks or private messages. The 
latter is measured separately (see below) . 

.Total hours online. 

.Total Number of Logins to the system. 
.Total number of private messages sent. 

.Number of different addressees for private messages sent during the 
last full month. This is a rough measure of the number of 
different communication partners with whom students are exchanging 
information online. 

By recording these data monthly, we could aggregate to obtain 
the total for the whole course, and could also examine the extent to 
which these measures of activity changed during the course. 

Other Types of Data 

In addition to standard questionnaires, the monitored data on 
participation, and grades on te.ts and the final grade for the 
course, several other types of data were gathered. 

Institutional Data 

During the 1986-87 academic year, measures of general verbal and 
mathematical ability (the SAT's) and level of academic performance 
(the Grade Point Average) were obtained from college records for each 
student, if the student agreed and signed a formal release. 

Feedback from Faculty 

An online conference for faculty, messages exchanged with the 

project director, and two day-long face-to-face faculty workshops 

82 



were used to exchange information about experiences conducting 
classes in the virtual classroom. Each faculty member also produced 
a description of their experiences in teaching online. This feedback 
from faculty, along with direct observation of the online classes, 
was used to generate the mostly qualitative, data that served as the 
basis for the guide to teaching online included in Volume 2 of this 
report, and was also drawn upon for sections of this volume. 

Interviews with Students 

Personal or telephone interviews were conducted with ten 
students. Most of these students were selected from a list of 30 
students who had given the most positive or the most negative ratings 
of VC on the post-course questionnaire, or who had dropped out and 
had not responded to the "dropout" questionnaire. A few "moderately 
negative" or "moderately positive" students were included in the 
personal interview sample in order to try to fill in the spectrum of 
reactions. The purpose of the interviews was to probe the reasons 
underlying the students' evaluations, and to explore the full context 
of experiences and circumstances which resulted in their opinions of 
the Virtual Classroom. 

MEASURING THE VARIABLES 
Many of the independent and dependent variables in this study 
are fairly simple and straightforward, such as age or gender, and 
v;ere measured with single questions on the questionnaires. Others 
•;neasure complex concepts, and were conceived from the beginning as 
composed of a number of dimensions, represented by a series cf 
questions. 

For all courses in all modes, a set of post-course questionnaire 
items was used to measure student perceptions of general 
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characteristics of the course content, the quality of the 
instruction, and course outcomes. An additional extensive set of 
items was used to measure student perceptions of the nature and 
quality of the online courses as compared to traditional courses » 
The first two sets of dependent variables (items dealing with course 
content and quality of the teaching) will be treated only in terms of 
a combined index in this study, since they were not conceived of as 
being substantially influenced by mode of delivery. The two sets of 
variables measuring course outcomes and VC ratings will be treated 
both individually, and in combined indexes. 

Constructing Indexes 

Many of the conceptual constructs being used in this study are 
multi-dimensional. It is more valid to use several items, each 
measuring a slightly different aspect of the variable, and then 
combine them, rather than relying on one question. In building these 
indexes, items were included in the questionnaires that appeared to 
have "face validity." That is, conceptually, they appear to measure 
some attitude or behavior that is included in the concept. After the 
data were collected, these intended scales were subjected to an item 
analysis to see if they were indeed correlated. A reliability 
analysis was conducted, which computes Cronbach's Alpha as an overall 
measure of the reliability of the composite measure. In this 
procedure, (provided by SPSSX but not by SPSS-PC), each designated 
component is left out of the total index and the Alpha level computed 
for an index without the item included. In arriving at the final 
induxes, we omitted items that did not correlate well with the index 
as a whole, and/or items which substantially lowered the Alpha value 
if they were included. 

Composite independent variables include the Personal Efficacy 
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and Interpersonal Control scales devised by Paulus and Christie 
(1981) for measuring a person's perceived "sphere of ccntrol." since 
the standard scale items and scoring were used, these scales are not 
included here; the items included can be seen in the Appendix, in the 
section of the pre-use questionnaire labelled "Images of Yourself." 

The set of items on "current feelings about using computers" 
were combined into an index of "Compviter Attitudes" (Table 2-3), The 
same items were repeated on the post-course questionnaire, with that 
index labelled as "Computer Attitudes-2 . " Similarly, the items an 
"expectations about the EIES system" were combined into an "EIES 
Expectations" index (See Table 2-4) . 

In the Computer Attitudes index, an item on perceived 
reliability of computers was originally included. It did not 
correlate well with the other items, anci lowered the reliability of 
the scale, so it was omitted. Apparently, people who otherwise have 
positive attitudes towards computers may nevertheless feel that they 
are unreliable. 

Indexes formed by combining items from the "course rating" and 
"instructor rating" portions ol the post-course questionnaire are 
shown in Tables 2-5 and 2-6. Because all of these items were worded 
the same way on the questionnaires, with "1" or "strongly agree" the 
most positive response, and "5" or "strongly disagree" the most 
negative, scores were not reversed on any items in constructing the 
index. This does result in indexes for these two constructs for 
which the highest total scores correspond to the worst ratings. Key 
course rating questions with high inter-correlations, chc:.<.:n from 
both the "Characteristics of the Course" and the "Course Outcome*^" 
section, were included in the Course Rating index. All of the items 
on the instructor were included in the Instructor Rating Index. 
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Multiple items measuring the course outcomes of increased 
interest in the subject matter and increased ability to synthesize 
material were combined into INTEREST and SYNTHESIS indexes (see Table 
2-7). The other items in the post-course questionnaire section on 
course outcomes were used individually. 

One interesting point to note about the Collaboration Index 
(Table 2-8) is that we had initially included an item in the 
"individual vs. group learning" section of the questionnaire which 
had the student rate the degree of competitiveness among the students 
in the class. This item was not highly correlated with the other 
items that we thought indicate collaboration, such as making friends 
and working cooperatively. Apparently, collaborative work can 
proceed >/ithin a competitive environment. One can assume that what 
happens when a competitive situation is perceived is that the 
students collaborate to form a team that can compete more effectively 
than an individual. 

Four of the items asking the students to directly compare the VC 
with the TC were used for a composite "vc OVERALL" index (Table 2-9). 
The item on preferring traditionally delivered courses was omitted 
because it was used only in the spring, and its inclusion lowered the 
number of cases too much. 

Measuring Writing Improvement 

All Upsala freshmen produce a "writi ^q sample" in an examination 
setting upon entering the college. This .s a response to an essay 
question. A different writing sample is then collected at the 
beginning of the Spring term. 

Both "writing samples" are holistically graded by faculty 
members^ who are trained in a "norming procedure" to consistently 
grade each essay as a whole on a 1 (totally incomprehensible) to 10 
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(excelXent) scale. After norming with samples from each set of 
essays, two judges grade each student essay, if there is more than 
one point difference in the scores a.*5signed, the essay is graded by a 
third judge. The two scores are averaged (or in the case of 
inv^onsistent ratings, the two raost similar scores are averaged.) 

Because of the nature of the norming procedure, it would be 
expected that the overall rlistribution of scores assigned in the 
Spring, after the Freshman writing course has been completed by 
students, would not be very different from that in the fall; in both 
cases, the students were being compared to one another. However, if 
the techniques useo in one particular section of the course are more 
effective than those used in others, then there ought to be a 
difference in the amount of change in scores, with the scores in the 
more effective section showing more improvement than average, it was 
planned to compare change in writing scores for the section that used 
VC with that in the approximately 14 other sections. 
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Table 2-3 



ITEMS IN THE COMPUTER ATTITUDES INDEX 

For each of the following pairs of words, please circle the 
response that is closest to your CURRENT FEELINGS ABOUT USING 
COMPUTERS. For instance, for the first pair of words, if you 
feel computer systems in general are completely "stimulating" to use 
and not at all "dull," circle "l"; "4" means that you are 
undecided or neutral or think they are equally likely to be 
stimulating or dull; "3" means you feel that they are slightly more 
stimulating than dull, etc. 



DULL-1 [R] 

Stimulating 12 3 4 
23% 24% 21% 21% 



5 6 7 Dull 
5% 2% 3% 



X SD 



2.8 1.5 



DREARY- 1 [R] 

Fun 12 3 4 

22% 27% 23% 15% 



5 6 7 Dreary 
8% 2% 3% 



2.7 1.5 



DIFFICULT-1 [R] 
Easy 12 3 4 

7% 15% 18% 27% 



5 6 7 Difficult 
16% 12% 5% 



3.8 1.6 



IMPERSONAL- 1 [R] 
Personal 12 3 4 

6% 10% 13% 36% 



5 6 7 Impersonal 
11% 1J% 11% 



4.2 1.6 



HELPFUL- 1 
Hindering 



12 3 4 
4% 2% 5% 15% 



5 6 7 Helpful 
16% 31% 27% 



5.4 1.6 



UNTHREATENING-1 
Threatening 12 3 4 

4% 6% 6% 26% 



5 6 7 Unthreatening 
12% 21% 26% 



5.0 1.7 



INEFFICIENT-1 [R] 

Efficient 12 3 4 5 6 

38% 30% 15% 10% 2% 2% 



7 Inefficient 
2% 



2.2 1.4 



OBLIGING-1 
Demanding 



1 2 3 4 5 6 
12% 12% 13% 40% 11% 8% 



7 Obliging 
4% 



3.6 1.5 



UNDESIRABLE-1 [R] 
Desirable 12 3 4 
25% 26% 16% 23% 



5 6 7 Undesirable 
3% 3% 4% 



2.8 1.6 



Notes: [R] indicates item was reversed for scoring 

Range » 7 (least favorable) to 70 (most favorable) 
Alpha= .82 
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Table 2-4 



Items Comprising the "EIES Expectations" Index 

Indicate your expectations about how it will be to use this system by 
circling the number which best indicates where your feelings lie on 
the scales below. 



EASY-1 

4% 6% 
: 1 : 2 
Hard to 
learn 



FRIENDLY-1 

4% 7% 
: 1 : 2 
Impersonal 



NOT FRUSTRATING-l 

4% 10% 
: 1 : 2 : 
Frustrating 



PRODUCTIVE- 1 

2% 1% 
: 1 : 2 
Unproductive 



14% 
3 



25% 
4 



19% 
5 



20% 
6 



(Mean=4.5, Std Dev= 1.6) 



8% 
3 



24% 
4 



28% 

5 



20% 
6 



(Mean= 4.6, Std Dev= 1.5) 



16% 
3 



24% 
4 



21% 
5 



21% 
6 



(Mean= 4.3, Std Dev= 1.6) 



5% 

3 



18% 
4 



24% 

5 



34% 
6 



(Mean= 5.3 Std Dev= 1.3) 



11% 
7 : 
Easy to 
learn 



9% 

7 : 
Friendly 



9% 

7 : 
Not 
frustrating 



16% 
7 : 
Productive 



EFFICIENCY-1 [R] 

Do you expect that use of the System will increase the efficiency of 
your education (the quantity of work that you can completa in a given 
time)? 



19% 21% 
; 1 : 2 
Definitely 
yes 



14% 24% 15% 

3:4:5 
Unsure 



5% 2% 
; 6 : 7 : 

Definitely 
not 

(Mean=3.2 Std Dev= 1.6) 



QUALITY-1 [R] 

Do you expect that use of the System will increase the quality of 
your education? 



21% 22% 
1 : 2 
Definitely 
yes 



18% 25% 6% 

3:4:5 
Unsure 



4% 3% 
6:7: 

Definitely 
not 
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(Mean« 3.0 Std Dev= 1.6) 



RESENT- 1 

I resent being required to use EIES for this course. 



4% 3% 6% 19% 

: 1 : 2 : 3 : 4 : 
Definitely Unsure 
yes 

(Mean^ 5.5 

OVERALL- 1 [R] 

Overall, how useful do you expect 

23% 27% 20% 19% 

: 1 : 2 : 3 : 4 : 
Very 
Useful 

(Mean= 2.8 



7% 17% 43% 

5:6:7: 

Definitely 
not 

Std Dev« 1.7) 



the System to be for online classe 

6% 3% 2% 

5:6:7: 

Not useful 
at all 

Std Dev= 1.5) 



EXPECTED TIME 

While you are part of an online course, how much time in the average 
week do you foresee yourself using EIES in relation to your 
coursework? 

(1) 4% Less than 30 minutes 

(2) 12% 30 minutes to 1 hour 

(3) 43% 1-3 hours 

(4) 29% 4-6 hours 

(5) _Z1 7-9 hours 

(6) 5% 10 hours or more 

Notes: Range = 9 (worst expectations) to 62 (highest) 
Cronbach's Alpha= .82 
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Table 2-5 

ITEMS INCLUDED IN THE COURSE RATING INDEX 



WASTE OF TIME (R) 

This course was a waste of time SA A N D SD 

COURSE OVERALL 

How would you rate this course over-all? 

(1) Excellent (2) Very good (3) Good (4) Fair (5) Poor 

MORE INTERESTED 

I became more interested in the subject SA A N D SD 

LEARNED FACTS 

I learned a great deal of factual material SA A N D SD 

CONCEPTS 

I gained a good understanding of basic concepts SA A N D SD 
CENTRAL ISSUES 

I learned to identify central issues in this field SA A N D SD 

COMMUNICATED CL^IARLV 
I developed the ability to communicate clearly 
about this subject SA A N D SD 

(R) INDICATES ITEM WAS REVERSED FOR SCORING 
RANGE= 7 (BEST) TO 35 (WORST) 
ALPHA= .88 
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Table 2-6 
THE INSTRUCTOR RATING INDEX 



WELL ORGANIZED 

Instructor organized the course well SA A N D SD 

GRADING FAIR 

Grading was fair and impartial SA A N D SD 

ENJOYS TEACHING 

Instructor seems to enjoy teaching SA A N D SD 

LACKS KNOWLEDGE (R) 
Instructor lacks sufficient knowledge 
about this subject area SA A N D SD 

IDEAS ENCOURAGED 

Students were encouraged to express ideas SA A N D SD 
PRESENTED CLEARLY 

Instructor presented material clearly 
and summarized main points SA A N D SD 

OTHER VIEWS 

Instructor discussed points of view 
other than her/his own SA A N D SD 

PERSONAL HELP 

The student was able to get personal 
help in this course SA A N D SD 

INSTRUCTOR BORING (R) 
Instructor presented material in 
a boring manner SA A N D SD 

HELPFUL CRITIQUE 
Instructor critiqued my work in 
a constructive and helpful way SA A N D SD 

TEACHER OVERALL 

Overall, I would rate this teacher as 

(1) Excellent (2) Very good (3) Good (4) Fair (5) Poor 

(R) indicates item scoring was reversed for the scale 

Range= 11 (best) to 55 (worst) 
Alpha= .88 
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Table 2-7 

Components of the INTEREST and SYNTHESIS Indexes 



Index of Increased INTEREST in the Subject 

MORE INTERESTED [R] 
I became more interested in the subject SA A N D SD 

DID ADDITIONAL READING [R] 
I was stimulated to do additional reading SA A N D SD 

DISCUSS OUTSIDE [R] 
I was stimulated to discuss related topics 
outside of class SA A N D SD 

[R] indicates response values reversed for index scoring 
Ranges 3 (least interest stimulated) to 15 

Alpha= .66 



Items included in the SYNTHESIS Index 

CENTRAL ISSUES [R] 
I learned to identify central issues in this fie]d SA A N D SD 

GENERALIZATIONS [R] 
My ability to integrate facts and develop 
generalizations improved SA A N D SD 

RELATIONSHIPS [R] 
I learned to see relationships between important 
topics and ideas SA A N D SD 

Range= 3 (low synthesis) to 15 
Alpha= .80 
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Table 2-8 

ITEMS COMPRISING THE "COLLABORATION" INDEX 



I developed new friendships in this class [R] 
I learned to value other points of view [R] 



SA A N D SD 
SA A N D SD 



Individual v,3. Group Learning 
Some courses are essiantially a very INDIVIDUAL experience; contact 
with other students does not play an important part in your learning. 
In other courses, communication with other students plays a dominant 
role. For THIS COURSE, please circle the number below that '^eems to 
be what you experienced. 



Individual 
experience 



3 



The help I got from other students was [R] 



12 3 
Crucially important 
to me 



4 



Students in my class tended to be 



6 

Group 

experience 



5 6 

Useless or 
misleading 



1 2 
Not at all 
cooperative 



Extremely 
cooperative 



How often did you communicate with other students outside of class, 
by computer, "face-to-face" or on the telephone? 



1 

Never 



Constantly 

Items marked R reversed for scoring 
Range =6 (least collaboration) to 34 (most collaboration) 

Alpha= .74 
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Table 2-9 

ITEMS COMPRISING THE "VC OVERALL" INDEX 



INCREASE QUALITY (R) 

Did use of the System increase the quality of your education? 

sl,:2:3:4:5:6:7: 
Definitely Unsure Definitely 

yes not 



NOT CHOOSE ANOTHER 

I would NOT choose to take another online course. 

! 1 : 2 : 3 ; 4 ; 5 : 6 : 7 : 
Strongly Strongly 
^9^^^ Disagree 

BETTER LEARNING (R) 

I found the course to be a better learning experience than normal 
face-to-face courses. 

Jl:2:3:4:5:6:7: 
Strongly Strongly 
Agree Disagree 

LEARNED MORE (R) 

I learned a great deal more because of the use of EIES. 

Jl:2:3:4:5:6:7: 
Strongly Strongly 
Agree Disagree 

(R) INDICATES ITEM WAS REVERSED FOR SCORING 
RANGE = 4 (WORST) TO 28 (BEST) 
ALPHA= .85 
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DATA ANALYSIS PLANS 
Variations by Mode and by Course 

As described previously, a quasi-experimental factorial design 
varying mode of delivery for five courses is at the heart of the 
design of this study. This basic design is supplemented by data 
collection on several other courses under various delivery modes, in 
order to increase the number of subjects for analysis and the related 
probability of obtaining statistically significant results. 

After obtaining univariate data on all independent, intervening, 
and dependent variables, each will first be analyzed using a one-way 
analysis of variance by mode, and separate analyses of variance by 
course and by "school" (Upsala vs. NJIT) . 

Bivariate correlations will be obtained for each independent or 
intervening variable vs. each dependent variable, for all VC 
students, for all students in traditional sections, and for all 
students combined. 

The next step will be a series of two-way analysis of 
variance ('anova') procedures to look for interaction: course by mode; 
course by first vs. second offering online; and mode by school. For 
these analyses, which will have very unequal N's and missing groups, 
we will use the SAS "General Linear Models" analysis of variance, 
which provides tests of hypotheses for the effects of a linear model 
regardless of the numta-sr of missing cells or the extent of uneven 
distribution of subjects (see User's Guide: Statistics, 1982, SAS 
Institute) ♦ 

Multivariate Analysis 
We are particularly interested in trying to untangle "cause and 
effect" with an experimental design that does not randomly assign 
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subjects to treatments, and in ^ ,..:'! - - .ferences in treatments 
(modes) may be confounded with b. ■ differences that are associated 
with educational outcomes. For instance, if we observe that there 
are differences among courses in such characteristics of students as 
previous Grade Point Average and SAT scores, which are measures of 
ability, and if the courses are also delivered in different modes, 
statistical methods can be used to pull out the relative importance 
of these factors. 

For each of the dependent variables or combined indexes of 
primary interest, we will select variables for multiple regression, 
based on observed significant bivariate relationships. 
We may also try introducing covariates into ANOVA's of course by 
mode. 



SUMMARY 

A dualistic evalution plan uses a quasi-experimental design to 
examine the issue of statistically significant differences in 
outcomes which are related to mode of delivery as it interacts with 
other variables. The research plan also utilises qualitative 
methods, including course reports by instructor and interviews with 
students, to explore in depth the behavior and attitudes which 
underlie these statistics, particulary for especially excellent and 
especially poor outcomes. 

The core quasi-experimental design employs matched sections of 
four courses, one section conducted totally in the Virtual Classroom 
environment, and one section conducted totally in the Traditional 
Classroom environment. This yields a basic 2 (mode) by 4 (courses) 
design. In order to obtain a much larger sample of students and a 
broader range of applications for both statistical and qualitative 
analysis, the design of the study was expanded in many ways. We 
added courses offered in a "mixed" mode, partially (at least 25%) VC 
and partially TC. We included post-graduate courses offered by three 
educatiui. . institutions to remote students, for which there is no 
"control" section meeting face-to-face. We also repeated several of 
the online courses a second time. 

Data collection methods included pre-and post-course 
questionnaires, motitor data for online activity, test scores and 
course grades, participant observation, instructor case reports, and 
interviews with students. Questionnaire items measuring subjective 
assessments of course effectiveness were drawn from widely-used 
instruments for measuring teaching effectiveness. Many of the 
dependent variables are multi-dimensional; indexes constructed for 
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these variables combine the answers to several related items from the 
post-*course questionnaire. 
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CHAPTER 3 
IMPLEMENTATION PROBLEMS 

Before reporting the results of this project, it is necessary to 
provide the context for these results. We will describe some of the 
problems which arose in implementing the Virtual Classroom for 
totally online delivery of undergraduate courses for credit, for the 
first time. As should be expected, Murphy's Law reigned supreme. 
Particularly during the first semester, when the quasi-experimental 
design oi matched online and face-to-face classes was carried out, 
there were many problems which deleteriously affected the online 
courses. In subsequent semesters, many of the problems were 
lessened, if not solved, and the results began to improve. 

One implication of our experiences is that other institutions 
should "start small." That is, start with only one or two courses 
online, and build from there. Vlith a fall semester set of offerings 
that included eight different completely or partially online courses 
and five "control" classes, spread over two campuses, we found 
ourselves in the situation of being unable to deal adequately with 
all of the minor crises and glitches that occurred. 

Recruiting and Enrolling Students 

The ideal student for the Virtual Classroom would be mature in 
terms of motivations about learning (seeking to learn as much as 
possible rather than to do as little work as possible) ; informed 
about the characteristics of this mode of. delivery; and the owner of 
a PC and modem at home (in order to maximize their access) . The 
ideal faculty member at an institution offering such courses would be 
informed about the advantages and disadvantages of VC delivery in 
order to advise prospective students, and supportive of a new means 
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to deliver education to students who might benefit from it. The 
ideal university bureaucracy would be flexible and have good internal 
communications, so that steps could be taken to assure ease of 
implementing an enrollment decision by a student once that occured. 
In fact, students, faculty, and administrators are likely to be 
re: stant, if not resentful or hostile, towards such an educational 
innovation, which they may perceive as a threat or an imposition. 

In the Spring of 1986, a full-page description of the Virtual 
Classroom experiment was developed. The plan was to include it as a 
page in registration materials at Upsala and NJIT, and to footnote VC 
courses with references to this information. The information 
included a provision that the student must speak to the faculty 
member in charge of the course to review the consent form, and sign 
and turn in such a consent form in order to register for the course. 
This information page was included with Upsala registration 
materials, which is provided to about 2000 students each semester. 

At NJIT, because of the expense, it was ruled that this full 
page of information could not be included in the registration 
information that was sent to thousands of enrolled and prospective 
students. Instead, each VC course carried two lines, "experimental 
course delivered via computer; see instructor for information." 
However, the campus newspaper carried the full information as a 
"front page" article. The registrar's office stated that procedures 
would be developed to make sure that students did not register for 
the course without a signed consent form. 

By August, pre-enrollment figures were dismal at both schools. 
There was one student enrolled for Introduction to Sociology at 
Upsala; three for Introduction to Computer Science at NJIT. By 
erecting barriers to enrollment, even potentially interested students 



were dl/acouraged. These barriers were inadvertently quite effective 
at NJIT. We discovered this when students who had intended to enroll 
in a VC section told the instructors that they had been iitformed that 
the VC section was closed, so they had enrolled in another section 
instead. Investigation of this mystery revealed that the registrar 
had decided to handle the consent form in the following manner. 
Capacity for the course had been set at zero; therefore, when a 
student tried to register, she or he would be told that the section 
was closed and that they would have to see the instructor for 
permission to register. However, the assistants actually present at 
registration did not know the special circumstances for why the 
computer was showing the sections as "closed." They simply told 
prospective students that the section was closed. As soon as this 
situation was discovered, the capacity was reset at 30, with the 
result that students began registering without understanding what it 
was that they were registering for. They simply would not take the 
trouble to seek out the instructor, as suggested in the registration 
material. Since instructors have only a few office hours a week, and 
students usually allocate just an hour or two to register for a 
semester, this is quite understandable. 

When the dismal enrollment situation was discovered in August, 
posters and flyers were prepared and distributed on both campuses. 
The poster listed all VC sections ?nd had a pocket for the flyers. 
There was a separate flyer for each course, with other VC courses 
available listed on the flyer also. The color was bright yellow. The 
posters were put near registration areas, in classroom buildings, and 
in bookstores and dormitories. 

In addition, at Upsala registration, the project director 
visited each faculty member advising students, explained the project, 



distributed brochures, and made a plea for them to "advise in" 
students who might benefit from this approach. 

The result was adequate numbers of students registered, but in 
many cases, these students were either totally ignorant of the 
experimental nature of the mode of delivery (having simply registered 
for an open section, without bothering to find out or perhaps even to 
notice the statement about "delivered via computer") ; or unsuited for 
this mode of delivery. For instance, a number of the students 
registered in the online section of Introduction to Sociology were 
ice hockey players. The project director advised two of these 
players when they attempted to register. The ice hockey players 
reported that their team met in the chapel basement, which was also 
the location for registration. They saw the poster and flyers there. 
Their coach took it as a way out of a scheduling dilemma. It seems 
that the team could only "get the ice" for practice from 1 pm until 4 
pm — five days a week. It was impossible for most students to find a 
full schedule of classes within these limitations, since they also 
could not take classes at night, when games were scheduled. The 
coach noticed from the posters and flyers that the VC section did not 
meet at any specified time, and therefore would not conflict with 
other courses, and advised any player who needed another course to 
sign up for it. These students had come to college largely to play 
hockey rather than for c.cademic reasons; they basically had no 
interest in Sociology but simply "needed a course;" and they attended 
other classes in the mornings and then went straight to hockey 
practice. After attending the initial training session, most of them 
signed on little or not at all. 

Soliciting in the Chapel " Advertising and recruiting students for 
specific courses is simply not done in academia. Thus, our posters 
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and flyers and personal communications were considered "unfair 
competition" by many faculty members. On both campuses, outrage was 
expressed at the means used to recruit students for the VC sections. 

At Upsala, the Project Director was accused in a meeting of the 
Educational Policies council of "soliciting students — in the chapel, 
no less." Questions were raised about the project's being illegal (in 
the sense of not following college regulations for course approvals) 
and unwise. Many members of the EPC felt that anything delivered via 
computer could not be as effective as a traditional course, and that 
educational quality was being endangered. Though in the past, EPC 
approval had been required only to introduce a new course, many 
members felt that this means of teaching was so radically different 
from their concept of "teaching" that approval should have been 
sought in order for the experiment to be offered. These same members 
indicated that they probably would not have given such approval. 
Though the Dean's approval for the project had been secured/ their 
reaction was that the Dean should not have approved the project and 
should have brought it to them for approval. 

During the same week in September, the project director received 
an irate call from a representative of the Organizational and Social 
Sciences department at NJIT. This department offers Introduction to 
Sociology at NJIT. They had been asked if they would offer one 
section online, but had declined. Upsala and NJIT have 
cross-registration agreements, whereby a student at either school can 
register for a course at the other. On all of the course brochures, 
other VC sections were listed. Therefore, for instance, Upsala 
students were informed that they could register for Introduction to 
Computer Science online, and NJIT students were informed that 
Introduction to Sociology, offered by Upsala, was available to them. 
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The OSS representative was angry and outraged, and implied that 
we could be stealing their students. This was unfair competition. 
Moreover they had not approved the course offered by Upssala for 
credit at NJIT. 

I explained that any NJIT student who tried to enroll for the 
Upsala course would have been required to check with his advisor and 
obtain approval for this course before enrolling, in fact, no NJIT 
student had requested enrollment. This latter fact mollified the OSS 
faculty meipber. However, he indicated that he felt that the approval 
of the OSS department should have been sought ahead of time, before 
listing this course as available to NJIT students? and that it was 
very, very unlikely that such approval would have been given. 

Despite the publicity that so roused the ire of faculty members 
on both campuses, many students showed up at the first VC session for 
many of the courses with no idea what they had signed up for. Thi.i 
theme comes out in several of the interviews with students included 
in the Appendix, particularly for students who felt negatively about 
the means of delivery. They simply did not see the material included 
in the registration information or the posters and flyers and 
newspaper articles available throughout the school. Though they were 
offered the oppurtunity to transfer to another section, they 
generally stated that the alternative section was scheduled at an 
inconvenient time. They started their training with a negative and 
resentful frane of mind... and in many cases, their attitudes slid 
downhill from there, since they were surprised and/or angry during 
the training session, they did not even hear some of the relevant 
information. For instance, all training sessions included a 
discussion of where and how to obtain u nodem and a special telephone 
line, if they had a PC at home but no modem. Students who were 
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"inadvtesrtsnt enrollees" tended not to hear or to remember havi 
heard this information* 



Inadequate Equipment 

Computer-Mediated Communication depends on many different pieces 
of equipment; if any one of them fails, the student is "shut out" of 
the "classroom." There is the central conferencing system itself, 
which may have hardware or software failures; its communications 
hardware and software for accepting incoming traffic from various 
sources; the telephone lines and/or packet network system through 
which the user reaches the system; and the micro, modem, 
communications software, and printer at the user's and. Our 
implementation was severely inadequate in terms of providing 
sufficient equipment at the user's end, and we also had some serious 
limitations with EIES. 

Ideally, every student taking a course partially or completely 
online would have a micro and a modem at home and/ or at work, and 
could dial in anytime. At the very least, there should be adequate 
access to high-quality and compatible equipment on a campus offering 
such courses. Such wis not the case, particularly at Upsala. 

Practically no Upsala students had microcomputers. On campus, 
there was a motley and inadequate collection of equipment. We had 
anticipated a major donation to the project from IBM, but they 
pleaded a change in financial resources vs. needs for their own new 
facility for corporate technical training at Thornwood, New York, and 
reneged, in the Upsala microcomputer laboratory, there was one ideal 
piece of equipment— An IBM PC-XT with a hard disk, 12 00 baud modem 
with Smartcom software, and 1200 baud printer that was reliable. We 
also had three Radio shacks that had no hard disks and completely 
different communications software; plus a shared printer for all 



three that only operated at 3 00 baud. There were three Apples with 
modems; they had still different communications software. Moreover, 
the apple configuration did not support continuous printing while 
online; the user had to continuously print one screen at a time. In 
addition there were a few 300 baud 'dumb' printing terminals spread 
around the campus; access procedures using this equipment were 
different than those required for use of the microcomputers, which 
further confused the students. 

To make matters worse, the operating budget of the Upsala 
microlab was such that it could only stay open about 50 hours a week, 
instead of a desirable minimum of 12 hours a day, six days a week. 
The result was that many students found it very difficult to match 
their need to use equipment to 'attend' their classes with the 
limited opportunities available. As will be seen from data presented 
later in this report, the Upsala students did not spend a great deal 
of time online — at least partially because access was so inadequate. 
(These access difficulties are described in more detail in Bob 
Meinke's report on the Introductory Sociology course at Upsala, in 
the appendix to volume 2) 

At NJIT, reshmen and sophomores had been issued their own PC's. 
However, they were not issued modems or printers, and many were not 
^'illing to buy them for this course. In the Virtual Classroom 
laboratory at NJIT, there were only seven micros, and only one of 
these with an attached printer. Students without micros at home 
needed to use an awkward and time-consuming "remote print" facility 
to get printouts. In the regular microcomputer laboratories, the 
administration refused to provide connections to EIES. Their 
statement was that the labs were already overcrowded, and they did 
not have the facilities to add connections to the local area network 
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for these machines. Thus, many of the NJIT students ended up on dumb 
CRT's placed in a big hallway, sending remote prints to a fast 
printer several floors below. This is hardly convenient or optimal 
access. 

Problems reported by students who did had micros and modems at 
home included difficulties with tying up their phone lines for hours 
at a time, and with lack of adequate documentation for communications 
software. One of the best communications software packages, 
SMA.RTCOM, is expensive. Instead, students made use of a variety of 
"shareware" or inexpensive programs with less functionality. We 
could not even tell them how to use much of this software to connect 
to EIES, since we had never seen it ourselves. 

Ideally, students should be supplied with a common piece of 
communications software, with the access numbers and parameters 
already set on their diskette. The shareware program "PROCOMM" is 
now available; if we had it to do over again, we would make diskettes 
of this software for all students with micros to use. 

A related problem was with student assistants, who were supposed 
to be available to keep the labs open and to help online students. 
Many of them proved unreliable for various reasons. Their priorities 
were elsewhere. For instance, if they had an exam or an assignment 
due in a course, they just didn't show up for their hours, and 
students found locked doors on the microlab. One assistant at NJIT, 
who had been scheduled for 15 hours a week of the time the lab was to 
be open, went to Taiwan for one month in the fall and another in the 
Spring, because his parents died. Our project staff was so small 
that we had no "backup" personnel to cover consistently when such 
events occurred. 

EIES itself is running on a minicomputer that is not very large 
m£ ion 



or powerful by today's standards. It slows noticably when more than 
about 30 users are online simultaneously, which tended to occur 
during the initial training sessions and at midday on weekdays. It 
can accept only limited numbers of users coming in through each 
possible channel: local area network at NJIT, 300 baud local, 1200 
baud local, and TELENET. The local area network access lines and/or 
the 1200 baud dialup lines were sometimes saturated during this 
experiment, forcing the students to try another access method or wait 
on a queue for a free line. In addition there was one serious crash 
during the fall semester, which came at the very worst time: during 
the last week of classes, when everything was "due." The EIES disks 
had filled up, and it took about two days to straighten out the mess 
and delete some unnecessary files. This was very frustrating and 
disruptive for the students, needless to say. (Note: We had been 
requesting additional storage capacity for over a year; the purchase 
order was not approved until its neccessity was demonstrated by the 
system coming to a complete halt. Such mechanisms for determining the 
true need for additional hardware resources are probably not unusual 
in universities, where there is competition for limited hardware 
budgets . ) 

Unfinished Software 

For a variety of reasons that will not be described in detail 

here, the actual signing of the contract for this project did not 

occur until November of 1986; meanwhile, the project supposedly 

started in January 1986. The start of software development was 

postponed while the question of whether the whole project was a "go" 

or "no go" was at issue. As a result, the special software which we 

had intended to have completed fell about six months behind schedule. 

Only an incomplete and very "buggy" version of the branch activities 
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was available at the beginning of the fall. The Personal TlilES 
graphics package was not completed until almost the end of the 
Spring. 

Perhaps the decision should have been "no go." However, it was 
not possible to postpone the experiment, since academic offerings are 
scheduled an entire year in advance. The choice was to proceed with 
unfinished special software tools, or to cancel the entire project. 
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Resistance to Collaborative Learning 
Most students are used to instructional designs that are based 
on either completely individual activity, or competition. The 
widespread practice of "grading on a curve" emphasizes competition 
and penalizes students for helping one another, when faced with an 
instructional design which calls for them to work with others in a 
cooperative or collaborative manner, particularly if they are 
expected to play a "teacher-like" role such as giving criticism of 
draft papers, many students are resistant. They may also feel that 
any grading scheme that makes their performance and grade dependent 
on collaborative work with others is "unfair." Finally, many students 
apparently place little value on the opinions of their peers. 

This attitude of little regard for or interest in communication 
from other students was apparent among some students at the very 
first training session, when asked to practice using the system by 
entering comments for one another, they were impatient about reading 
material contributed by their peers, asked how tc break the output, 
and wanted to know how to go straight to the assignments and lectures 
contributed by the instructor. If this attitude toward communicating 
with and working with their peers persisted, they were unlikely to 
feel positively about the Virtual Classroom approach. 

Materials in Interviews 2 and 4 are relevant to this 
generalization. Note that the student in Inteicview 2 complains about 
VC being "self-study." When asked about his reactions to the 
contributions of the other students, he said, "I usually just blew 
off the other class members' comments and went straight to the 
professor's lecture. I wouldn't say that the other students' 
comments were a waste of my time; I just didn't read them." 
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Similarly, in interview 3, a very negative student had no interest in 
even looking at material contributed by other students. 

On the other hand, students who worked hard on collaborative 
assignments and then were "let down" by other group members also had 
very negative feelings, at the time. As a student in Organizational 
Communication who had finished her part of a group activity on time 
put it, "I don't think it's fair that those of us who worked so hard 
to get our information on the computer have to suffer for those who 
don't bother to get their assignments in on time!" A subsequent 
message assuring her that she would receive an "A" for her excellent 
and lengthy contribution did not make her feel a whole lot better 
about it. She messaged back about still feeling disappointed when 
she came to the lab looking fon/ard to reading contributions by 
others, only to find that the "others" had not appeared. The 
students who were late completing their parts of an online 
collaborative activity were the same ones who were chronically late 
doing traditional individual handwritten or typewritten assignments. 
In the latter case, however, their tardiness did not interfere with 
the learning of other students, whereas in a collaborative online 
assignment, it did. 

Another problem is getting students to offer constructive 
criticism to one another; this is an unfamiliar role. In the 
partially online writing course at Upsala, for instance, Mary 
Swigonski required each student in a writing group to respond to 
specific questions on on another's draft essays. On a particular 
writing exercise, they might have been asked to suggest a better 
opening, suggest a better organization, and to suggest a better 
closing. Each student was to use these comments to produce an 
improved final draft. Dr. Swigonski reports that in responding to 
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these questions on each peer's essay, she could not get the students 
beyond "being nice" to one another. They felt comfortable saying 
what was good about the draft essay, but did not feel comfortable 
offering criticism, she encouraged the students to use pen names, but 
reports that they still did not feel comfortable making critical 
comments . 

In future studies, the reasons for students' reluctance to offer 
constructive criticism to one another should be investigated with 
unstructured interviews focussed on this issue. Perhaps, for 
instance, students feel that their peers would be upset by critical 
remarks, even if offered in the context of suggestions for 
improvements. They may be reluctant to risk causing hurt or anger 
which would negatively affect their relationships with one another. 
Perhaps they feel unqualified to make such suggestions, especially in 
a "public" forum. Or, alternatively, they may feel that by helping 
one another o\t, they might be negatively affecting their own grade, 
if the class is graded on a curve. Finally, the observed problem may 
be related to student grade-oriented motivations. In the tipsala 
writing course, students were required to say something about each 
peer's draft essay in the small writing groups. However, they were 
not graded for the quality of their suggestions. In many courses, 
instructors have observed that the students at these two colleges 
allocate their effort roughly in proportion to its importance for 
their grades. Since anything above "zero effort" counted the same, 
they may simply have taken the rational time-allocation choice of 
making the minimal effort needed to maximize their grades. If the 
reasons for the failure of students to offer constructive criticism 
on drafts are understood, then it may be possible to change the 
social dynamics in future online classes. 
^ 113, 
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Electronic Pranks 

For some students, CMC represents a fascinating opportunity for 
mischief, minor and major. It is inevitable that students will be 
tempted to abuse the medium. 

As Keenan (1987) points out, on the public and private BBS 
systems, some people are posting information that goes beyond the 
obscene and annoying and becomes truly dangerous and/or criminal. 
For instance, a BBS allegedly operated by a Ku Klux Klan chapter 
gives the names, addresses, and license plate numbers of KKK 
"enemies," including rabbis and suspected FBI agents. A BBS in 
Calgary contained plans for causing the city's Light Rail Transit 
train to crash; other entries have included things from directions 
for making an atom bomb or drugs to credit card numbers and 
instructions for "phone freaking." 

Nothing quite this dire happened during the Virtual Classroom 
experiment. Students were ;arned orally and in one of the first 
messages they received that irresponsible behavior would result in 
loss of their accounts, just as disruptive behavior in a traditional 
classroom would result in their being asked to leave the class. They 
were specifically instructed not to send messages, anonymous or 
otherwise, to anyone who was not in their class and whom they did not 
know. Of course, some ignored this and sent personal and sometimes 
obscene messages to strangers they saw online. We have no idea how 
often this happened without complaint from the "victim," but in over 
half a dozen cases, there were complaints, and steps were taken to 
warn the offending student and/or to remove the account, depending on 
the severity of the breach of standards for acceptable student 



conduct* 

Some students figured out how to steal an ID and use it to 
misbehave without much threat of exposure and punishment; they 
obtained other people's accounts from users who were careless about 
not protecting their passwords. In one case, several fraternity 
"brothers" of a sick student "helped him out" by signing online for 
him while he was in the hospital, and took the opportunity to send 
obscene messages to whatever females happened to be online at the 
time — under their fraternity brother's name, of course. 

Another student went this one better. He/she observed an 
instructor's password during a demo; the instructor evidently did not 
change his code after the demo. In the middle of the night, the 
perpetrator got online using the ID of the instructor; sent a series 
of extremely objectionable propositions to just about everybody 
online; and also posted several comments in public conferences, under 
the instructor's name, making scandalous remarks about the purported 
behavior of the President of the University. All of the latter were 
erased by the next morning; EIES users are for the most part a 
self-policing community. One of the recipients immediately sent a 
message of complaint about "Professor X's" message to the system 
monitor and user consultants; the system monitor then used his 
emergency privileges to delete all the conference comments and freeze 
the account. However, this should serve as an important cautionary 
tale for instructors and others. DO be careful to protect your access 
code! Use a temporary code for all demonstrations, and then change 
your access code immediately afterwards « 

In sum, it is inevitable that the freedom and new opportunities 
for communication offered by CMC will be abused by some immature 
and/or irresponsible students. Policies must be developed which 



provide guidelines, and describe the consequences of unacceptable 
behavior online. These must be communicated clearly to the students, 
and enforced. 
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Relaxing Experimental Controls 
The initial quasi-experimental design called for the "matched" 
sections of four courses to be "the same" in every way except that 
one section would be completely online (meeting face-to-face only for 
training, the midterm, and the final) and the other section would be 
completely face-to-face. They were to have the same content and the 
same assignments. The assumption that this could be done without 
cripling the potentials of the medium or raising ethical issues 
turned out to be incorrect. In fact, in all of target courses, 
adjustments had to be made. 

Even before the semester started, the instructors pointed out 
that to require the same assignments in the matched sections would 
severely limit their ability to make use of the unique 
characteristics of the medium. The VC supports collaborative 
assignments and in-depth discussions, whereas the TC does not. So, 
though the offline reading assignments and the exams remained the 
same, the assignments given students were quite different for the two 
modes. This was true even for the Upsala statistics course, for 
instance, where the online section began with students filling out a 
questionnaire in the class conference, and then using the data 
provided by the other class members to carry out a statistical 
analysis. The offline section did this assignment using a 
pre-supplied data set. 

The instructor for the NJIT statistics course found that many of 
the students wanted to work together in parallel, taking the 

opportunity to ask questions of her or the other students 

Q 116 ^ rjn 



face-to-face, while working online. She scheduled a once a week, 
two-hour session when she was available in the NJIt microlab. About 
a third to a half of the class seemed to show up each week 
(unfortunately, we did not keep records of which ones). Generally, 
there would be periods of one or two students working silently at 
each of the terminals in the lab; periods where subgroups would be in 
animated discussion around a terminal, pointing at the screen; and 
short periods when several or all of them were conferring with the 
instructor about a question raised by the online material. We had 
not anticipated this "group lab" adaptation of the medium, but the 
instructor felt that it worked well for her and her students. 

In computer science, th. instructor found that the students 
could read through and understand the written version of his lecture 

•rial in a much shorter time than was required to cover the same 
tile irial by talking and listening and taking notes. Therefore, he 
s emented the online section by adding seme additional activities 
e cterial which was not included in his traditional section. 

' Sociology, the online assignments were totally different than 
tho^i the matched face-to-face section. These online assignments 
invol. .dying and discussions. However, the midterm exam was 

based mai on the textbook. There were many more failures on the 
midterm in the online section. The instructor felt that perhaps this 
was not fair to the students, since they had been tested on material 
which was not similar to the assignments they had been doing. 
Therefore, two optional face-to-face exam review sessions were held, 
and those who attended were given the opportunity to retake the 
midterm. This incident underscores the impossibility of complete 
"matching." The two media are suited to very different types of 
learning and assignments, and it does not make sense to try to test 
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the students using the same examination. Nevertheless, we stuck 
rigidly with the use of the same midterm and final in all courses for 
this study. 
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Summary 

The implementation of virtual classroom was far from optimal. 
Problems included: 

.Recruiting sufficient numbers of students for the experimental 
online sections. 

.Opposition from faculty members who believed that the medium would 
fail to adequately deliver college-level courses, and/or that it 
would be unfair competition which would decrease enrollments in 
their courses. 

.Failure to adequately inform all students enrolled in the 
experimental sections of the nature of the educational experience 
in which they would be involved, despite explanations in 
registration material, campus newspaper articles, flyers and 
posters V 

.Inadequate amounts and qua"* ity of equipment for student access. 

.Limited capacity of the central host (EIES) , which was sometimes 
saturated. 

.Unfinished software tools to support the Virtual Classroom, 
including the absence of the graphics package that had been 
considered so important for some of the courses. 

.Resistance by some students to collaborative learning, 

.Deliberate misbehavior by some students. 

.Impossibility of rigid experimental control which "holds everything 
constant" except the medium of course delivery. 

These problems interacted. For instance, we had initially 

anticipated only four courses involved in the experiment. Partially 

because of the low enrollments in the experimental sections, many 

other courses were added to the study. Each additional course had its 

own unique problems and demands, which added to the overload on the 

limited staff for the project. We were working undar a contract that 

specified tight deadlines for completion of phases and 

"deliverables." It ..ould have been far better to spread out the 

implementation over a longer period of time. However, the rigidity 

of the academic ca]endar and scheduling conventions (whereby courses 
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and teaching assignments are scheduled as much as a year in advance) 
and of the project contract requirements made this impossible. 



1? <■■,/- 



ERIC 



CHAPTER 4 

WHAT HAPPENED IN THE VIRTUAL CLi^SSROOMS? 

In this Chapter, we will review the level of activity which 
occurred in the Virtual ClassroCiUS and the students' ratings of and 
coininents about their experiences. We will examine how the VC mode of 
delivery seems to have affected educational process and outcomes, on 
the "average" and as it varied among courses. 

The Appendix includes data on the overall means and frequency 
distributions of responses to the pre- and post-course 
questionnaires. These results will be referred to in sections of 
this chapter. Rather than constantly repeating the full text of 
questions, each one has been given a short label, which also appears 
in the Appendix. 

OVERALL (AVERAGE) VC RESULTS 

Reasons for Taking a VC Course 
For all students in all modes, among the most important 
motivations for enrolling in a course are that the course is required 
for graduation (56% reported this reason as "very important"), or 
required for a major (47%) . Job-related interests or general 
interest in the topic also characterize a substantial nun\ber of 
enrollees (32%) . In deciding whether to sign up for a traditional 
vs. a virtual classroom section, two additional motivations may come 
into play: curiousity about (or attraction to) the medium, and 
convenience. 

There were significant differences among courses in the extent 
to which mode-related motiva\:ions characterized the students' 
reasons for taking a particular course and a particular section of a 
course. For the two "distance education" courses included in the 



study, greater convenience and curiousity about or attraction to the 
medium was a very strong factor (see Table 4-1. Distributions for 
partially online courses with no matching section were omitted, since 
these students had no choice of section or mode) . These factors 
also played an important role for the totally online courses at NJIT. 
At Upsala, they were important for many or most of the students who 
enrolled in Sociology online, but not for the students in the 
statistics course. 
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Table 4-1 
Reasons for Taking VC Courses 
% Choosing "Very Important" 



Job General Required Required Instructor No Curious More 

Interest Major Grad Reputation Choice Convenient 
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Sample Interaction in the Virtual Classroom 

One way to begin to understand what happened in the Virtual 
Classroom is to look at a sample transcripts of parts of courses. 
Several excerpts are included as an Appendix to Volume 2 of this 
report. In this volume, we will include part of what happened during 
one week in Introductory Sociology, a course which illustrates many 
of the problems as well as many of the potentials of using the VC 
mode of course delivery. 

There is a great deal of variation in perceptions of 
characteristics of the Virtual Classroom, both among courses and 
among students in the same course. However, some "central 
tendencies" include the following: 

.Greater candor, among those who participate; and 
.A tendency towards procrastination. 

Both of these tendencies are illustrated in the Exhibit from a 
module in the Introductory Sociology course. The instructor reports 
that the students seemed to feel more at ease about revealing 
personal experiences in relating examples to apply and illustrate 
sociological comments. Certainly, many of the responses in the 
exhibit relate to very personal aspects of the students' lives. 
About half of the students chose to use their pen names, and the 
other half did not. The half that signed their assignments with 
their names do not seem any less candid than the half who used the 
privacy protection provided by a pen name. 

Some of the entries are so poorly written that it is difficult 
to understand them. This should not be attributed to typing errors; 
many of the Basic Skills essays hand written by Freshmen show the 
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same types of pervasive grammatical errors. As we will see later in 
this chapter, these students had fairly low levels of skill for 
college-level work, as measured by SAT scores and grade point 
averages for other courses . 

The excerpts also show the tendency of students to put off 
assignments and other forms of online participation. The first 
assignment was due by Midnight on a Tuesday night. Several of the 
entries were made after dinner on that evening, since the students 
did not have computers at home or in their dormatories, this meant 
that they had to make a special trip to a computer terminal in the 
evening. 

The close times of several of the items suggest that the 
students were in fact in the laboratory together. It was a common 
practice for two or three students in an online course to develop a 
"buddy system" and sit next to each other and talk over things that 
were coming across the screen, and help one another with the 
mechanics of using the system or the contents of the material. 
Though this was supposedly not allowed during qiiizzes, it undoubtedly 
occured then too. 
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Exhibit 



EXCERPTS FROM INTRODUCTION TO SOCIOLOGY 

Note: Only minimal editing of student comments has been dene, in 
order to preserve the tendency towards mistakes in grammar and 
spelling that pervade many of the entries. A name in quotes means 
that the student chose to enter a response with a pen name. Other 
names have been removed. 

The instructor's comments have been greatly shortened, in order to 
give just the essence of the material to which the students were 
responding. 

:C2039 CC148 Robert Meinke (Bob M,1571) 10/ 9/86 10:08 AM L:145 
KEYS: /ROLE STRAIN/ ASSIGNMENT #9/ 

(YOU MAY WANT TO MAKE A PRINTOUT OF THIS LONG MINLECTURE AND 
ASSIGNMENT) 

Your text briefly discusses the topic of ROLE STRAIN. I would 
like to amplify that discussion because role strain is one of the 
most prevelant sources of discomfort in people's lives, probably also 
in yours. 

ROLE STRAIN: The difficulty experienced by an individual in 
meeting the expectations of his or her roles. 
Role strain has two major causes: 

ROLE CONFLICT: Conflict due to incompatible demands of one's 
roles. 

ROLE AMBIGUITY: Discomfort because what is expected of one in 
certain roles is not known or not clearly understood. 

(over 100 lines of "minilecture" deleted here) 

ROLE STRAIN: ASSIGNMENT #9 

ENTER AS A CONFERENCE COMMENT. DUE: TUESDAY MIDNIGHT, 10/14. 
USE YOUR PEN NAME. USE KEY: ROLE STRAIN/ ASSIGNMENT #9 

1) Describe in detail an experience of real role strain that you 
have experienced soT^etime in your life. 

2) In sociological terms, what was its cause? Was it due to: 

a) role conflict 

-a role incompatible with your personality 

-conflict between the role demands of two different statuses 
-conflict between two roles in one role set 
-conflict between the demands within one single role 
-conflict with a role partner over the meaning of that role 

b) role ambiguity 

-because the role was a new undefined role 
-because the expectancies of the role were rapidly changing 
-because you were entering a new life status which you didn't 
feel prepared for 

3) How did you try to resolve the strain? 

a) compartmentalization 

b) hierarchy of obligations 
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c) banded together with others to change the social definition 
of the role 

d) renegotiated the role definition 

e) left the status 

f) chose an emotional outlet to escape 

:C2039 CC173 "M0NIQUI2" 10/13/86 11:31 AM L:18 
KEYS: /ROLE STRAIN/ ASSIGNMENT #9/ 

AN EXAMPLE OF ROLE STRAIN THAT I AM EXPERIENCING NOW IS BETWEEN 
SCHOOL AND WORK. I WORK FOR A MAJOR CORPORATION WHILE GOING TO 
SCHOOL FULL-TIME. HOWEVER, MY EXPLOYER WOULD LIKE ME TO PUT IN MORE 
HOURS THAN I DO NOW. THE STRAIN THAT I FEEL IS THAT I KNOW I NEED A 
FOUR- YEAR DEGREE TO ADVANCE IN THE COMPANY, YET THEY EXPECT ME TO 
WORK MORE WHILE IN COLLEGE. WITHOUT THE DEGREE, I WILL NEVER GET 
ANYWHERE IN THE COMPANY. 

2) THE CAUSE OF THE ROLE STRAIN IS ROLE CONFLICT- CONFLICT 
WITHIN THE DEMANDS OF ONE SINGLE ROLE. 

3) I TRIED TO ESTABLISH AN HIERARCHY OF OBLIGATIONS TO RESOLVE 
THE CONFLICT. I WILL NOT GO TO COLLEGE LESS THAN FULL-TIME, SO ALL 
OF MY SPARE TIME IS DEVOTED TO V "jRKING. THIS WAY I CAN GAIN WORK 
EXPERIENCE, AND, HOPEFULLY, BE HIRED AT A HIGH LEVEL AFTER I GET MY 
FOUR-YEAR DEGREE. 

:C2039 CC177 "MONEY" 10/14/86 11:47 AM L: 12 
KEYS: /ROLE STRAIN/ ASSIGNMENT 9/ 

ONE EXPERIENCE OF ROLE STRAIN WAS AS AN EMPLOYEE OF UPSALA 
COLLEGE. THE PROBLEM WAS ROLE AMBUGUITY, I CAME INTO A JOB WHOSE 
DUTIES WERE NOT CLEARLY DEFINED. IT WAS ALSO AT THE TIME OF A CHANGE 
IN S^'^ERVISOR. I WAS HIRED BY AN ACTING DIRCTOR, BUT WHEN I REPORTED 
TO ^ I FOUND A NEW DIRECTOR. THE JOB DESCRIPTION WAS NON-EXISTENT 

AND THE NEW DIRECTOR NEVER TOOK THE TIME TO DEVELOP ONE. I TRIED TO 
RESOLVE THE CONFLICT BY ESTABLISHING A HIERARCHY OF OBLIGATIONS, AND 
ALSO BY RENEGOTIATINGWITH MY SUPERVISOR WHAT THE ROLE SHOULD BE. I 
FINALLY LEFT THE POSITION FOR A MORE STABLE ONE. 

:C2039 CC179 (Name, Nickname, IL) 10/14/86 1:48 PM L:24 
KEYS: /ROLE STRAIN/ ASSIGNMENT #9/ 

ONE OF THE MOST DIFFICULT ROLE STRAIN THAT I HAVE EXPEERIENCED 
IS WHAT IS EXPECTED OF A YOUNG WOMEN. THIS HAPPEN TO ME A COUPLE OF 
YEARS A GO. I REAL I,Y ENJOY RACKETS BALL AND MY MOTHER AND BOYFRIEND 
KNEW THIS. THEY DID NOT SEEM TO MIND ME PLAYING, BUT ONCE THEY FOUND 
OUT THAT I HAD JOIN A CLUB WHICH HAD RACKET BALL TOURNMENTS THE I DAL 
OF ME PLAYING WAS WRONG, AND I WAS CONSIDERED OUT OF PLACE. MY MOTHER 
SAID THAT IT LOOK BAD FOR A LADY PLAYING BALL WITH MEN, OR COMPETEING 
WITH MEN IN A SPORT. MY BOYFRIEND GAVE ME LITTLE TALKS ABOUT HOW 
UNLADY LIKE IT IS PLAYING AGAISTED MEN THEN HE TOLD ME THAT 
PRJISPERATION DOES NOT HELP WOMEN BUT HINDER THEM. A THIS WAS A 
CONFICT OF ROLE, THE TYPE OF ROLE CONFLICT IS ROLE AMBIGUITY, HE AND 
MY MOTHER DID NOT WANT TO ACEPT THAT ROLE EXPECTANCISE ARE RAPID LY 
CHANING. 2 2) IN SOCIOLOGICAL TERMS, THE CAUSE WAS B) ROLE AMBIGUITY 
BECAUSE THE EXPECT ANCIES OF THE WERE RAPIDLY CHANING. 3) I TRIED TO 
RESOLVE THE STRAIN BY RENEGOTIATED THE ROLE DEFINITION OF WHAT IS 
EXPECTED OF A YOUNG LADY. 
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:C2039 CC181 (Name,ENicknaine, ID) 10/14/86 8:04 PM L:16 
KEYS: /ROLE STRAIN/ 

A DAUGHTER TO A MOTHER IS AN EXArfPLE OF ROLE STRAIN. DAUGHTER 
WHICH IS ME AS A TEENAGER GROWING INTO AN ADULT. I HAVE AN DIFFERENT 
OPINION ON THINGS THAT MY MOTHEii CANNOT RELATE TOO. I GUESS THERE IS 
AN REBELLION STAGE WITHIN THE TEENAGE YEARS. MY MOTHER STATES HER 
OPINION AND EXPECTS ME TO AGREEE AS A GOOD DAUGHTER SHOULD DO. THIS 
CAUSES A GREAT CONFLICT. 

HER ROLE OF A DAUGHTER IS ONE WHO LISTENS AND OBEYS TO WHATEVER 
SHE MAY SAY. 2.) THE CAUSE WAS DUE TO ROLE CONFLICT. A ROLE 

INCOMPATIBLE WITH MY PERSONALITY CONFLICT BETWEEN THE DEMANDS WITHIN 
ONE SINGLE ROLE AND CONFLICT WITH A ROLE PARTOVER THE MEANING OF THAT 
ROLE. 3.) I TRIED TO RESOLVE THIS STRAIN THROUGH RENEGOTIATION. I 
WOULD LISTEN TO HER OPINIONS AND TAKE THEM INTO CONSIDERATION BUT 
ALSO HAVE HER TO LISTEN TO MY OPINIONS AS WELL. WITH BOTH MAYBE WE 
COULD COME TO SOME REASONABLE RESULT. ; 

:C2039 CC183 (Name, Nickname, ID) 10/14/86 8:26 PM L:8 KEYS;/ROLE 
STRAIN/ ASS. #9/ 

1. I EXPERIENCED ROLE STRAIN WHEN MY MOM REMARRIED AND MY 
STEPFATHER -FATHER WAS INTRODUCED INTO MY HOME. I HAD TO ASSUME A 
NEW ROLE AS A STEP-DAUGHTER WHICH INCLUDED ASKING HIM FOR PERMISSION 
TO GO OUT OR TO USE THE CAR. ASKING FOR MONEY WHEN I OR MY MOM 
DIDN'T HAVE ANY, ETC. 2. IN SOCIALOGICAL TERMS MY ROLE STRAIN WAS 
CAUSED BY ROLE AMBIGUITY. 3. I RESOLVED THIS ROLE STRAIN BY 
RENEGOTIATING MY ROLE AS A STEP-DAUGHTER WITH MY STEP-FATHER. HE IS 
MY MOTHER'S HUSBAND AND I WILL GIVE HIM RESPECT FROM TIME TO TIME 
BUT THEN I WILL LOOK UPON HIM AS A FATHER IN CERTAIN SITUATIONS. 

:C2039 CC184 (Name, Nickname, ID) 10/14/86 8:28 PM L:19 
KEYSJ/ROLE STRAIN/ASSIGNMENT#9/ 

I EXPERIENCED ROLE STRAIN WHEN I ENTERED BUCKNELL UNIVERSITY AS 
A FRESHMAN. I HAD NO PREVIOUS PROBLEMS IN ASSUMING THE ROLE AS A 
STUDENT IN HIGH SCHOOL (ROLES INCLUDED BEING SOCIABLE AND STUDIOUS, 
WHICH LEAD TO ACADEMIC ACHIEVEMENT) , BUT I EXPERIENCE DIFFICULTY AT 
BUCKNELL BECAUSE I COULD NOT ASSIMILATE THE COLLEGE LIFE. AS A 
RESULT, I WAS UNABLE TO BE SOCIABLE, STUDIOUS, AND ACHIEVE ACADEMIC 
SUCCESS. MY GRADES; OF COURSE SUFFERED DRASTICALLY, AND I BEGAN TO 
FEEL SOCIALLY CONFINED. SUPPORT WAS NOT GIVEN TO ME BY OTHER 
STUDENTS AND BUCKNELL FACULTY. AS A STUDENT I WAS ENTITLED TO THIS 
SUPPORT. 

ROLE AMBIGUITY CAUSED MY ROLE STRAIN, FOR I WAN NOT PROPERLY 
PREPARED FOR LIFE AS A COLLEGE STUDENT. I HAD NO FORMER EXPERIENCES 
TO RELY ON PREPARATION FOR THIS NEWLY ACQUIRED OR ACHIEVED STATUS. 

I RESOLVED MY ROLE STRAIN BY LEAVING THIS STATUS. I DROPPED OUT 
OF COLLEGE AFTER THE FIRST SEMESTER OF MY SOPHOMORE YEAR VOWING NEVER 
TO RETURN TO SCHOOL, ESPECIALLY BUCKNELL UNIVERSITY. OBVIOUSLY, I 
DID NOT KEEP THIS VOW. I NOW FEEL THAT THE TWO YEARS I HAD TAKEN OFF 
FROM MY FORMAL EDUCATION HAS ENABLED ME TO MKE A MORE MATURE 
APPROACH TO BEING A COLLEGE STUDENT. 
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student Perceptions of the virtual classroom 
In the following pages, we will suitunariiie students' reactions to 
their VC experience across all coursei that were offered totally or 
partially online. It must be kep": in mind, however, that "average" 
responses and reactions are obtained by combining results for courses 
which varied a great deal. 

Included in the Appendix are the complete distributions for 
responses to the post-use questionnaire on the items which asked all 
students who used the Virtual Classroom to compare their experiences 
to previous experiences in courses delivered entirely "face-to-face." 
These questions were 1 to 7 Lickert-type scales, with responses 
ranging from "strongly agree" to "strongly disagree." The responses 
from 1 to 3 were summed as indicating agreement, and those from 5 
through 7 as indicating disagreement. 

Convenience ; The majority (65%) felt that taking online courses 
was more convenient. Even those students who generally prefer 
traditional courses tended to comment on the advantages of being able 
to worK on the course at times of their own choosing. For instance, 
in the fifth interview in the Appendix, a student from the fall 
Statistics course at Upsala commented, 

I liked that I was independent and that I could go whenever 
I wanted to. And I like how the conferences were written down 
and I could get my notes. It also helps if you miss a day or 
two, because the computer always has your assignments there for 
you. 

Those with computers and modems at home were of course, most 
likely to appreciate the convenience. For example, in the j^ixth 
interview in the Appendix, a Management Lab student said. 

It's also good because there is easy access whenever you 

129 



^ 129 1 A ^ 

ERIC 141 



want. I have a modem at home. I can go on at, 3 o'clock in the 
morning. That's usually when I do most of my work. 

Themes related to the greater convenience and comfort of 

attending class online also appear in the comments offered by 

students about what they "liked best" about the virtual classroom. 

"Being able to do the assignments at my own pace and not being 

obligated to sit in a very confined classroom;" "the freedom;" "being 

able to put the information into the computer whenever it is 

convenient;" "flexible class hours," and "not having to go to class" 

are some of the attributes mentioned. 

More Work ; The majority (63%) disagree that they "didn't have to 
work as hard for online classes." The fact that most felt that they 
worked much harder also comes out in the interviews with students and 
the course reports from instructors. However, it should be noted 
that the instructors did not unanimously agree with the student 
perceptions that they were working harder for online courses. 

It is definitely true that the most enthusiastic students spent 
a great deal of time in their online courses. For example, a very 
positive student who participated in the Management Lab reports: 

I sign on every day. I usually spend about an hour; it 
depends how much other work I have. Sometimes as little as half 
an hour; sometimes two or three hours. Sometimes I sign on 
several times a day. I spend a lot of time online. I love it... 
I don't mind putting in the hours, the time just flies by. 

Irregular Participation ; Almost half (49%) admitted that when 
they became "busy" with other things, they were more likely to stop 
participating in an online course than to "cut" a traditional class. 
This is the flip side of self-pacing. Many students just did. not 
have the self-discipline to stick to a regular, frequent schedule of 
signing online and working. For instance, see the second studunt 
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interview in the Appendix. This student remarked, "I don't feel thac 

I have the self discipline for it. I don't have enough time in my 

day as it is. To sit down and make myself do something like that..." 

The students who did not participate regularly recognized that 

they were not able to get much out of the course by letting 

everything go until the last mirsute. For instance, a student who got 

a "D" in Computer Science got into the habit of staying lat« at work 

only one night a week to use the computer frum there. He explains 

his apparent inability to make time for regular and leisurely 

participation in the course as follov/s, (from Interview 9) : 

My downfall was in trying to minimize reading of the 
comments during the time I had to devote to it. I didn't read 
them on the screen, I printed them out and took them home. Then 
things would happen. I work long hours, I live alone and have 
to cook dinner.. I did look at a few of them... but I tried to 
do everything as fast as I could in order to maximize what I 
could finish during that one night. I tried to bring the 
paperwork home, but you bring home a book and often it does not 
happen... I read maybe 60% of it. 

As a result, instructors began devising strategies to force 
frequent signon, such as weekly quizzes due on a different day than 
the assignment, or raising the proportion of the grades allocated to 
online participation. (See, for instance, the course narratives in 
the Appendix of Volume 2 by the instructors for Introductory 
Sociology^ Computer Science, Statistics, and the Management Lab.) 

Incre a^d Intere st, Inv olveme nt , and Motivation ; For those who 

did participate, the level of interest and involvement tended to be 

high. 55% agreed that the fact that their comments would be read by 

other students increased their .motivation . 62% disagreed that the 

Virtual Classroom was "more boring" than traditional classes, and 56% 

agreed that they felt more involved in taking an active psrt in the 

course. The word "fun" was frequently used by those students who 

reported high levels of interest and involvement. 
Q 131 

ERIC ' 143 



Less Inhibition: The questionnaire item was worded negatively, 
in terms of feeling "more inhibited." 44% disagreed, and 29% 
perceived no difference between modes. This was obviously an aspect 
of online participation which varied a great deal among students and 
perhaps among courses, as a result of levels of writing skill, 
self-confidence, and the atmosphere established by the instructor. 

Sociology Instructor Robert Meinke reports, in his course 
narrative, that 

Online courses do encourage students to write better 
responses to their assignments. The fact that other students 
will read what they have written often stimulates more effort. 
I also found that students seem to feel more at ease about 
revealing personal experiences. The options that EIES provides 
of sending anonymous or pen name responses encourages the more 
shy person to express him or herself more openly. 

A Math 305 student (Interview i) said that he felt "more free" 
to say things online: 

I may seem gregarious, but I'm pretty shy. it's easier 
from here. Because it seems like one-on-one. 

Related to the general perception that the written word allows 

people to be somewhat more "free" in expressing themselves, is the 

feeling expressed by several students that the medium makes grading 

more "fair." A CIS student in interview lo remarked: 

All he knows is what you type. He can't be prejudiced 
against you based on the way you look... It's more fair this 
way. You're being judged really on your work, not on your 
personality. 

On the other hand, some students felt more inhibited, especially 
about asking questions that might expose them as "ignorant." while 
students might join in a discussion or a simulation, they were more 
reluctant to ask questions about the reading or a lecture. Some of 
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this reluctance may he due to a false assumption that they might be 
penalized for a "stupid" question. The Upsala statistics student in 
Interview 3 explains 



Sometimes you don't feel comfortable asking the teacher 
questions through the computer. In class, you can raise your 
hand, or you can ask questions aftur class. It is not as 
comfortable to ask a question online, so you don't ask... Maybe 
he will take off credits or something. Sometimes it is too late 
to put a question in- the assignment is already due. it's more 
personal when you see the teacher. 

Especial.i.y in the more technical courses, such as statistics and 
Computer Science, the instructors also experienced a difficulty in 
eliciting and responding to student questions and assignments online. 
For instance, Lincoln Brown explained the relative lack of instructor 
responses to student comments in his class conference as follows: 



Where students had problems, I sent them messages. 

While I plead' guilty to not providing positive feedback, 
note that there's not much vhich can be said about many of their 
comments. For example, when simply asked to look at a graph and 
comment on which bar is higher, they all mads s;oir.e appropriate 
but innocuous comment. 

And look at the timing problem I mentioned in the report. I 
gave an assignment on March 27th; the first solution was entered 
on April 6th; most cair^e in on April 15th (future taxpayers 
practicing with this deadline!) I had been collecting responses 
on paper as they came in, but didn't girade them or comment until 
after the due datfli ( a mistake on my part.) In a few cases 1 
believe I responded to each with a grade and a one~line comment 
via one of BJ's +quiss - related programis. 

I believe the whole idea of "coiiments" is fundamentally 
different in a math course and, say, a isociology course. Maybe 
Rose found it not to be so - I wish I had had time to follow her 
conference while mine was going on «• but probably most of the 
time there will be this difference. 



laSj^easM JJlterajstlon The majority of stvidentjs (58%) felt that 
they had better access to their professor in the Virtual Classroom. 
This interaction was al.30 more "friendly" and equalitarian tnan would 
be typical of the traditional classroom. For example, a Math 305 
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student said: 

She'll put a message in and say, "Have a great week...» 
Especially, if you have a message or a problem, she'll write 
back and say, "Hi there, how have you been? You have a problem 
with this..." It's really almost like talking on the phone. I 
try to send messages back the same way, real casual. It's not a 
strict teacher-student kind of thing. Because of her, you feel 
a lot closer, because it's so easy just to pop a question. 
She'll answer the next day, or whenever you come online. 
(Excerpt from "Interview 1" in the Appendix) . 

Opinion is more mixed about whether the Virtual Classroom led to 
more communication with other students in the class: 47% agreed, but 
19% perceived no difference between delivery modes on this criterion, 
and 32% disagreed. On related items, 55% agreed that the fact that 
their work would be read by other students increased their 
motivation; 59% found the comments made by other students to be 
useful? and 62% found reading the reviews or assignments of other 
students to be useful. 

Those who were most enthusiastic about the medium tended to 
value the contributions and comments of other students highly, and to 
enjoy reading them. Among the phrases that are used in describing 
what students "like best" about the Virtual Classroom (in response to 
the open-ended question on the post-course questionnaire) , students 
mentioned "Class participation," "Being in touch with other students 
constantly," "Working as a group and extended communications online," 
and "the openness- I liked to hear other students' ideas." A Math 305 
.•student reported (Interview 1) that the comments of other students 
were 

...entertaining. Some of those people have some witty 
commants. That makes the class more interesting. If you find 
that there are a lot of comments, then you get online just to 
see them. 

By contrast, a negative, student in the same course commented, "I 
Ufsually just blew off the other class members' comments and went 
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straight to tht. professor's lectures." a negative student in the 
Upsala statistics course refused to read anything written by 
students, and referred to student contributions as "junk." A 
classmate in the same course reported, however, 

Most of the students who made comments were the ones who 
really understood the class and they were about the lectures. 
And they were pretty helpful, especially when the homework could 
be checked. 

An Organizational Communication student commented as follows 
about the value of reading the comments of other students: 

r.^^^ ^ 5®^^ ^^^^ really helpful. It gave me another 

°" ^ ""^^ ^ see a point and 

they did, I was able to incorporate it into my thinking... it 

was really a good way of learning diffe; ant ideas. 

Inter-Item ^rrelations: We have reviewed responses to ll 
questions asking students for comparisons between the traditional and 
Virtual Classroom environments. Only one of the 55 inter-item 
correlation coefficients was particularly high: finding the comments 
of other students useful and reading the assignments of other 
students correlated at .70. The other dimensions were clearly 
distinct in the students' minds, in the sense that response patterns 
were different. For example, the next highest coefficient was .57, 
between increased convenience and whether the VC was more boring. 
Thirteen of the coefficients weje under .10. This suggests that the 
students did tend to read each of the statements carefully and 
responded to each one individually, rather than adopting en automatic 
"response set." 

Overall Subjective Evaluations by Students 
Use of the Virtual Classroom on EIES was more widely perceived 
as increasing educational quality (56% agreed and 22% saw no 
difference) as compared to traditional modes of delivery than as 
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increasing educational efficiency (44% agreed, and another 23% saw no 
difference in "efficiency," at least with the current system and 
hardware access shortcomings) . In terms of overall comparisons about 
whether the Virtual Classroom approach "provides a better learning 
experience than normal face-to-face courses," 47% agreed and 25% felt 
that it was neither better, overall, or worse; it was just different. 
Asked if they "learned a great deal more'' using EXES than they would 
in a traditional course, 45% agreed and another 27% neither agreed 
nor disagreed. Perhaps this item should have been worded as simply 
"learned more" rather than "a great deal more," since the proper 
response for a person who learned a little more is not obvious. 

However, on both thr,se items and on the negatively worded items, 
there are about 20% of the students who definitely did riot like the 
Virtual Classroom as well as the traditional classroom, as indicated 
by their choice of one of the two most negative points on the scales. 
In assessing the statement, "I would have gotten more out of a 
traditional course," 24% agreed and 56% disagreed. 26% agreed and 
64% disagreed with the conclusion, "I would NOT choose to take 
a.iOther online course." Thus, the mean and median responses on 
overall assessments of the Virtual Classroom experience tended to be 
positive, but there was a sizable minority who did not like it as 
well as the traditional classroom. Much of the remainder of this 
report will be devoted to analyzing the effects of characteristics of 
students and other variables which help to explain the variations in 
assessments and outcomes. 

Lvidence on Dropouts 
One of the most important behavioral indicators of dislike of 
the Virtual Classroom approach is the rate at which students drop 
courses offered via this mpde, as compared to the dropout rate for 
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similar courses offered offline. There definitely was a greater 
tendency towards dropout in VC sections. This seems to be related to 
the tendency of students with poor study habits and a lack of 
self-discipline to procrastinate, then realize that they are 
hopelessly far behind, and drop the course. (There may be a 
disproportionate tendency for students with many family and job 
obligations to elect a course via this medium in the first place, but 
this is only speculation) . 

Unfortunately, students who were not very reliable about 
completing their online work regularly and who dropped out of courses 
offered via this mode were also very elusive when we tried to get 
data from them. All "dropouts" were sent two copies of the special 
questionnaire prepared for them, with the second letter pleading the 
importance of having their responses. Only nine returned it; none 
from Upsala. All dropouts who did not return a questionnaire were 
called more for an interview. Only one could be contacted by phone; 
the others were never at home. Thus, the evidence we have is 
incomplete. 

Table 4-2 shows the results for the nine dropouts who did 
respond to the questionnaire. Some of the reasons, such as "family 
problems" and "had a similar course already" are not related to mode 
of delivery. Of the nine, three would not choose to take another 
course via this mode. Two of the nine agreed that they "did not like 
the Virtual Classroom approach." On the whole, then, the reasons 
given by dropouts who responded tended not to be strongly critical of 
the medium, but instead reflected the types of reasons given for a 
decision to drop any course. 
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Table 4-2 

Reasons Given for Dropping Virtual Classroom Courses 

Question: How important were each of the following factors in your 
decision to drop the course? 



Reason 



Health problems or 
personal problems 

The course was too hard 
for me 

The course was too much 
work 

I did not like the 
instructor 

The subject matter was 
boring or irrelevant 

I had too many other 
courses and needed to 
drop one (or more) 

I was doing poorly 

I did not like the 
"virtual classroom" 
approach 

I had too many outside 
demands (other classes, 
full-time work) 



Very Somewhat Not X 
Important Important Important 



SD N 



22% 
11% 



22% 
22% 
22% 

11% 
22% 

33% 



11% 

22% 



11% 
11% 



78% 2.56 0.88 

89% 2.78 0.67 

89% 2.89 0.33 

56% 2.33 0.87 

78% 2.56 0.88 

78% 2.56 0.88 

78% 2.67 0.71 

67% 2.44 0.88 

67% 2.33 1.00 



9 
9 



If I had the opportunity, I would register for another class which 
used the "Virtual Classroom" approach: 



11% 
1 

Strongly 
Agree 



22% 
2 

Agree 



22% 

3 

Don't 
Know 



0% 
4 

Disagree 



44% 

5 

Strongly 
Disagree 



MOST IMPORTANT REASON 

(1) 38% CONFLICTING DEMANDS 

(2) 12% SIMILAR CLASS 

(3) 12% FAMILY PROBLEMS 

(4) 25% TOO HARD 

(5) 12% DISLIKE INSTRUCTOR 
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VARIATIONS AMONG COUFSES 
"Course" is not a unidimensional variable, it innludes 
differences in type and level of subject matter; ditferences in type 
of use of the system (totally online vs. partially online courses); 
cognitive level of the students (mostly freshmen vs. upper classmen 
or graduate students); differences in teaching style and procedures; 
and is also confounded with differences in access to the system, 
since some courses were offered through Upsala, where equipment 
access was relatively poor. It is not possible to separate out which 
aspect of "course" may account for significant differences in 
outcomes among courses. But this much is clear: on almost every 
measure of process and outcome, there are substantial and 
statistically significant differences among courses. 

Variations in Student Ability, by Course 
In addition to differences among courses in the initial 
motivations of students, there were also differences in ability 
levels. We collected data on overall Grade Point Average and on SAT 
scores for those students included in the quasi-experimental research 
design. These data are shown in Tables 4-3 through 4-5. Note that 
the Introductory Sociology students in the online section were fairly 
weak students. Their average GPA was only 2.0 (the minimum average 
required for graduation) , and ijoth their verbal and math SAT scores 
were fairly low. In addition, there was a difference among the 
Upsala statistics sections. Those students in the fall VC section 
werti relatively good students with better Grade Point Averages. The 
Spring VC section students in the Upsala Statistics course, by 
contrast, were not particularly strong, and in fact had a Math SAT 
average just under 400 ♦ 
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Table 4-3 



OVERALL GRADE POINT AVERAGES OF STUDENTS, BY COURSE 

QUASI -EXPERIMENTAL DESIGN 



COURSE 



CIS 213 

MATH 305 

MANAGEMENT 471 

INTRODUCTORY 

SOCIOLOGY (150) 

STATISTICS 

(CC140Y) 



FALL 


FTF 


SPRING 


ONLINE 




ONLINiS 


2.9 


3.1 


2.7 


2.6 


2.8 


2.5 


3.1 


2.5 


2.8 


2.0 


2.5 




2.7 


2.2 


2.3 



Tabie 4-4 

MEAN SAT VERBAL SCORES, BY COURSE 
QUASI -EXPERIMENTAL DESIGN 

COURSE FALL FTF SPRING 

ONLINE ONLINE 

CIS 213 333 

MATH 305 375 

MANAGEMENT 471 454 

INTRODUCTORY 365 
SOC (150) 

STATISTICS A 427 
(CC140Y) 

SECTIONS WITH THE SAME LETTER SIGNIFICANTLY DIFFERENT 

P<.05, DUNCAN MULTIPLE RANGE TEST 



400 
455 
430 
361 



A 332 



444 
364 
435 



371 



Table 4-5 
MEAN SAT MATH SCORES, BY COURSE 
QUASI -EXPERIMENTAL DESIGN 

COURSE FALL FTF SPRING 

ONLINE ONLINE 

CIS 213 640 

MATH 305 590 

MANAGEMENT 471 580 

INTRO SOC (150) 409 

STATS (CC140Y) A 492 

SECTIONS WITH THE SAME LETTER SIGNIFICANTLY DIFFERENT 

DUNCAN MULTIPLE RANGE TEST 



580 
480 
542 
374 
A 346 



571 
458 
D73 



399 



Access problems and Activity Levels 
The less than ideal access conditions described in Chapter 3 
were reflected in post-course ratings of access problems. On a 
one-to-five scale, where "1" is "Serious Problem," and "5" is "Not a 
problem," those who responded with a 1 or 2 rating can be considered 
to have experienced difficulties. Overall, 22% said that access to a 
terminal or micro was a problem; 19% had problems with busy ports to 
EIES; and 33% complained of slow system response, ks would be 
expected, those problems were much more prevalent at Upsala. 

Differences in access problems, as well as in the mode of 
employment of the system, are reflected in Table 4-6, which shows 
monitor statistics measuring mean activity levels of students in the 
different courses. Activity levels varied tremendously among 
courses, with the highest activity levels occuring for the fall 
Computer Science course, and some very low levels of use for several 
of the Upsala courses where the system was used as an adjunct to 
face-to-face instruction. Consistently, both frequency of 
participation and total time spent online are much lower for the 
Upsala courses. 

Two points should be kept in mind in examining these data. One 
is that the Connected Education students were specifically coached on 
how to upload and download from their micros, in order to decrease 
connect time, and many of the NJIT students also used this technique. 
Secondly, the Upsala statistics course was only a "half-course" 
lasting seven weeks, including the orientation meeting and the final 
exam. Even adjusting the data for the statistics course for the 
length of time, the average participation was very low, especially 
for the Spring online course. On the other hand, it is apparent that 
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in some of the courses, such as the two online sections of Computer 
Science, the Spring Math 305 course at NJIT, and the spring 
Management Lab at NJIT, the average student was checking in almost 
daily, and sent many private messages in addition to participating in 
the class conference. 

Table 4.7 shows that the amount of participation in class 
conferences differed among the courses from a low of less than 50 
comments in the main class conference for the mixed luode courses at 
Upsala to almost 1000 comments in the spring management lab 
conference. The pattern of balance between instructor contributions 
and student contributions also differs markedly. The most technical 
of the courses— Computer Science and the two Math/Statistics 
courses — tended to be "teacher-dominated" in terms of the proportion 
of contributions, whereas the courses in "softer" subjects tended to 
have the majority of comments contributed by students. 
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Table 4-6 

DIFFERENCES IN MEAN [(PER STUDENT) J ACTIVITY LEVELS, 

BY COURSE 



COURSE TOTAL TOTAL TOTAL 

HOURS TIMES MESSAGES 

ONLINE SENT 

CIS FALL 74.8 143.0 43.0 

CIS SPRING 30.2 97.2 21.1 

MATH 305 FALL 25.2 58.3 20.9 

MATH 305 SPRING 44.9 80.3 14.7 

MANAGEMENT FALL 17.7 39.4 9.1 

MANAGEMENT SPRING 43.2 90.1 22.7 

SOCIOLOGY FALL 18.2 37.0 23.2 

STATISTICS FALL 7.9 25.2 8.2 

STATISTICS SPRING 5.5 16.3 4.5 

CONNECTED EDUCATION 13.0 41.7 8.1 

ORG. COMMUNICATION 14.0 30.7 9.0 

WRITING SEMINAR 8.3 14.4 2.5 

ANTHROPOLOGY 4.3 7.1 1.2 

FRENCH 8.0 20.7 4.2 

F 2.3 3.9 2.5 

p .01 .001 .01 
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Table 4-7 

PARTICIPATION PATTERNS IN CLASS CONFERENCES 



COURSE N STUDENT INSTRUCTOR TOTAL % COMMENTS % LINES 

COMMENTS COMMENTS COMMENTS INSTRUCTOR INSTRUCTOR 



CIS FALL 


17 


148 


242 


390 


Q 6 ^ 


•719? 


CIS SPRING 


21 


93 


173 


266 


65% 


73% 


MATH 305 FALL 


13 


55 


119 

A A a/ 


174 


Aft 9^ 




MATH 305 SPRING 


27 


366 


111 


477 


23% 


49% 


MANAGEMENT FALL 


23 


367 


56 


423 


13% 


11% 


MANAGEMENT SPRING 


32 


826 


173 


999 


17% 


17% 


SOCIOLOGY FALL 


17 


265 


115 


380 


30% 


64% 


STATISTICS FALL 


14 


70 


55 


125 


44% 


81% 


STATISTICS SPRING 


12 


45 


33 


78 


42% 


81% 


CONNECT-ED-1 


13 


330 


62 


392 


14% 


12% 


CONNECT-ED-2 


13 


310 


102 


412 


25% 


28% 


ANTHROPOLOGY 


12 


40 


19 


59 


32% 


18% 


WRITING SEMINAR 


18 


33 


6 


39 


15% 


21% 


ORG COMMUNICATION 


12 


58 


35 


93 


38% 


32% 


FRENCH 


8 


50 


11 


61 


.18% 


23% 



KEYS: 

N - Total number of students enrolled 

STUDENT COMMENTS - Total number of comments entered by students 
INSTRUCTOR COMMENTS - Number Of comments entered by the instructor 
TOTAL COMMENTS - Total number of comments 

% COMMENTS INSTRUCTOR - Percentage of comments entered by instructor 
% LINES INSTRUCTOR - Percentage of lines entered by instructor 
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Outcome Differences Among Courses 

We have seen that student characteristics and activity levels 
varied among courses- In looking at the results, there were 
statistically significant differences among courses for almost every 
dependent variable, as determined by a oneway analysis of variance, 
A few of these differences will be presented and reviewed here. 

Table 4-8 shows differences in courses on some of the indices of 
process and outcome. On the collaboration index, high scores 
correspond to higher levels of perception of collaborative or "group" 
learning. The highest levels of collaborative learning occurred in 
the Management course; it was also high for Organizational 
Communication, Business French, the online writing seminar, and Math 
305. The level of reported collaborative learning appears to differ 
much more among courses than among sections of the same course 
offered in different modes. 

For the Instructor Rating and Course Rating indices, high scores 
correspond to the least favorable ratings. Once again, differences 
among courses appear to be much larger than differences among 
sections of the same course offered via different modes of delivery. 
The only course for which there is a sigiixf leant difference among 
sections is the Introductory Sociology course, where the students 
rated the instructor and outcomes as better in the face-to-face mode. 
In the computer science course, by contrast, the instructor and 
course ratings are higher in the virtual classroom mode. There is 
also a tendency for Lome of the best ratings to occur for the second 
repetition of an online course by an instructor. 

In the following table (4-9) , results are shown by course for 
the items which deal with overall coiaparisons between modes of 




delivery, including the index "VC OVERALL" which combines four items. 
High values of this index are the most favorable. The best overall 
ratings are for the second offerings of the Computer Science and Math 
305 courses, and the Ontario Institute course, which was offered by 
an instructor experienced in this mode of teaching. The ratings for 
the Upsala freshman-level totally c line courses tend to be among the 
lowest. By contrast with the students in the three upper-level NJIT 
courses, these students tended to feel that online courses are more 
boring, to disagree that they were more involved, and to agree that 
they would not choose another online course. However, these ratings 
are not characteristic of the upper-level, partially online courses 
at Upsala. 

It will be noted that differences among courses are associated 
with differences between the two colleges. Much of this has to do 
with the poorer access conditions present at Upsala. As with course 
as a variable, "school" was significantly related to differences for 
most outcome variables. Table 4-10 shows some of these results. The 
Upsala students perceived the system as less "friendly" and less 
"convenient." They were less likely to feel that they communicated 
more with other students or the professor, or that they learned more. 




Table 4-8 

SUBJECTIVELY RATED OUTCOMES, BY COURSE 
MEANS AND ANALYSIS OF VARXANCE 

COURSE INSTRUCTOR COURSE COLU> 30IIATI0N 





RATING 




RATING 




IfiOEX 




INDEX 




INDEX 






CIS FALL FTF 


28.5 




17*8 




18.9 


CIS FALL ONLINE 


25.4 




14.3 




20.0 


CIS SPRING ONLINE 


20.6 




14 « 8 




18.9 


MATH 305 FALL FTF 


15.7 




13.6 




23.1 


MATH 305 FALL ONLINE 


14 . 8 




12 . 5 




22.1 


MATH 305 SPRING ONLINE 


19,2 




14.5 




21.7 


MANAGEMENT SPRING FTF 


21.4 




15.0 




25.3 


MANAGEMENT FALL ONLINE 


23 . 1 




16 . 7 




26.1 


MANAGEMENT SPRING ONLINE 


18.0 




13.9 




27.2 


SOCIOLOGY FALL FTF 


A 19.3 


A 


13.7 


A 


23.9 


SOCIOLOGY FALL ONLINE 


A 25.5 


A 


17.6 


A 


17.2 


STATISTICS FALL FTF 


26.9 




19.0 




22.9 


STATISTICS FALL ONLINE 


25.8 




18.7 




21.0 


STATISTICS SPRING ONLINE 


25.9 




17,8 




20.2 


CONNLCT~ED 


25 . 0 




17 . 0 




19.1 


ONTARIO INSTITUTE 


19.0 




13.6 




22.6 


ORG. COMMUNICATION 


22.2 




15.2 




24,3 


WRITING SEMINAR 


18.4 




13.7 




23.4 


ANTHROPOLOGY 


18.6 




14,1 




20.9 


BUSINESS FRENCH 


20.8 




13,3 




24.6 


F 


7.7 




2.6 




5.3 


P 


.001 




.001 




.001 



A- The two sections are significantly different 
Duncan Multiple Range Test (p <.05) 

KEY: Instructor Rating Index Range = ii (best) to 55 (worst) 
Course Rating index Range = 7 (best) to 35 (worst) 
Collaboration index Range- 6 (least) to 34 (most 
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Table 4-9 

DIFFERENCES IN PERCEPTIONS OF THE VIRTUAL CLASSROOM, 
BY COURSE: MEANS AND ANOVA 



COURSE 


ONLINE 


MORE 


WOULD 


BETTER 


VC 




MORE 


INVOLVED NOT 


LEARNING 


OVER 




BORING 




CHOOSE 






CIS FALL 


4 . 8 


2.8 


5.1 


3.4 


19.4 


CIS SPRING 


5.7 


3 . 1 


5 . 7 


2.7 


20.5 


MATH 305 FALL 


4.6 


3.6 


4.3 


3 . 6 


17 . 0 


MATH 305 SPRING 


4.8 


3 . 5 


5 3 


3.3 


19.7 


MANAGEMENT FALL 


5.0 


3 . 0 


5 . 2 


3 . 4 




MANAGEMENT SPRING 


6.2 


2 . 0 




2 . 0 


23 . 0 


SOCIOLOGY FALL 


3.9 


4.4 


3.8 


4.6 


14 . 5 


STATISTICS FALL 


3.9 


4 . 4 


3 . 6 


5.0 


13.9 


ol/iiXbilCS SPRING 


3 . 9 


5.0 


3.6 


5 . 0 


14. 3 


CONNECT-ED 


5.5 


3.3 


6.7 


4.5 


18.5 


ONTARIO INSTITUTE 


6.3 


2.9 


6.3 


2 . 8 


21.5 


ORG. COM. 


4.2 


3.6 


4.1 


4.1 


15.4 


WRITirG SEMINAR 


4.1 


3.0 


4.1 


4 . 1 


16. 6 


ANTHROPOLOGY 


3.9 


4.0 


3.3 


4.9 


13. 6 


FRENCH 


3.5 


3.2 


4.5 


4.2 


16. 5 


F 


3.0 


2.7 


3.7 


3.7 


3.4 


P 


.001 


.001 


.001 


.001 


.001 


Key: l=Strongly agree, 


7=Strongly disagree 






"VC" Overall 


index may 


range 


from 4 (worst) to 


28 (best) 
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Table 4-10 



SELECTED SIGNIFICANT DIFFERENCES IN 
VIRTUAL CLASSROOM RATINGS, BY SCHOOL 



QUESTION 


UPSALA 


NJIT 




F 


P 


SLOW RESPONSE 


3.4 


3.7 


7. 


"2 


.008 


EASY TO LEARN 


4.4 


5.6 


19. 


77 


.001 


EIES FRIENDLY 


4.0 


5.4 


25. 


03 


.001 


EIES INCREASED QUALITY (R) 


4.2 


3.1 


12. 


76 


.001 


CONVENIENT (R) 


4.4 


2.6 


36. 


75 


.001 


COMMUNICATED MORE (R) 


4.4 


3.4 


9. 


92 


.002 


ACCESS PROFESSOR (R) 


4.0 


3.0 


9. 


91 


.002 


MORE BORING 


4.1 


5.3 


14. 


66 


.001 


MORE INVOLVED (R) 


4.1 


3.0 


16. 


87 


.001 


NOT CHOOSE ANOTHER 


3.8 


5.4 


21. 


46 


.001 


BETTER LEARNING (R) 


4.5 


3.0 


22. 


57 


.001 


LEARNED MORE (R) 


4.4 


3.2 


16. 


34 


.001 



Note: Items are 1 to 7 scales. Those with an (R) indicate 
that scoring is reversed, so that low scores are "better." 



Process and Outcome: Relationships at the Course Level 
A number of dimensions on which courses varied significantly 
have been displayed and discussed. One \»ay to pull this information 
together is to look at the extent to which rank ordering of courses 
on outcome measures is related to rank ordering on other variables. 
Some results of this analysis are shown in Table 4-11. 

The first thing to notice is that all the Upsala courses are at 
the bottom on the "VC OVERALL" index. In other words, outcomes were 
better for every single NJIT course than for every single Upsala 
course . 

A second noticeable tendency is that the "top three" courses in 
overall ratings were the second semester offerings of courses at 
NJIT; there is a consistent improvement with experience by the 
instructor for these courses. 

Looked at on the course level, with only 13 cases, student 
overall ratings of the Virtual Classroom are strongly related to 
amount of activity in their class conferences. The rank orders for 
average number of times each student signed online and for the total 
comments in the class conference are shown as examples. The courses 
with the best outcomes were those in which the students signed on 
frequently, and in which there was a lot of activity. (Which is 
cause, and which effect, is impossible to untangle with these data) . 

On the other hand, we totally failed to be able to explain 
variations in course outcomes ir. terms of any codable aspect of 
instructor behavior. An example is shown in Table 4-11 for a simple 
measure, the total proportion of comments by students. (A previous 
table showed the obverse, the proportion by the instructor) . We had 
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thought that classes in which the professor stimulated the students 
to do most of the writing would have better results than those in 
which many of the entries were by the instructor. However, even on 
this basic measure of process, there is no significant relationship. 
Several of the more "teacher-dominated" sections of courses, in Math 
and Computer Science, were among the highest ranking on overall 
student ratings of their VC learning experience. 



Table 4-11 

RANK ORDERS OF COURSES: PROCESS VS. OUTCOME 





VC overall 


Mean Times 


Total 


% By 




Index 


Online 


Conferevce 
Comments 


Students 


NJIT Management Spr (M) 


1 


3 


1 


3 


NJIT CIS Spring 


2 


2 


6 


12 


NJIT Math 305 Spring 


3 


4 


2 


5 


NJIT CIS Fall 


4 


1 


4 


11 


NJIT Management Fall (M) 


5 


6 


3 


1 


NJIT Math 305 Fall 


6 


5 


7 


13 


Upsala Writing (M) 


7 


12 


13 


2 


Upsala French (M) 


8 


10 


11 


4 


UDsala Org. Comm. (M) 


9 


8 


9 


8 


Upsala Sociology 


10 


7 


5 


6 


Upsala Statistics Spring 


11 


11 


10 


9 


Upsala Statistics Fall 


12 


9 


8 


10 


Upsala Anthropology (M) 


13 


13 


12 


7 



Key: M denotes a mixed mode course 
Spearman's Rho's: 

VC overall with Times online: 0.82, p=0.001 
VC overall with Total comments: 0.70, p=0.004 
VC overall with % by students: 0.11, p=0.36 
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SUMMARY 

Average subjective ratings of the Virtual Classroom by students 
are shown in Table 4-12, rank ordered from those items on which 
students were most enthusiastic or positive to those on which they 
were least positive. Among the attributes of the Virtual Classroom 
experience which are rated highly are increased access to the 
professor, increased interest and involvement, and being able to see 
othe*.- students' assignments. On the downside, students were more 
likely to procrastinate and stop actively participating online when 
they became "busy with other things," and they felt that VC requires 
them to work harder. 

There was a great deal of variation around these averages. In 
some courses, students were much more active and involved than in 
others. In addition, on almost every criterion, there was a 
difference between Upsala and NJIT, with NJIT students viewing their 
experiences more favorably. This ma;* be due both to the poorer 
equipmenL situation at Upsala; and/or to the fact that the Upsala 
courses that were totally online were freshman-level, whereas all the 
NJIT courses were at a sophomore or higher level. 
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TABLE 4-12 

Summary of student Perceptions ot the Virtual Classroom 
Characteristics Better Neutral Worse 



2.0 3 . 0-"' 4 . 0 5 

More From Traditional (R) 2.4 

Choose Another (R) 3.1 

(Not) More Boring (R) 3,2 

Others' Assignments Useful 3.2 

Better Access to Professor 3 . 3 

More Involved 3 . 3 

Comments Useful 3.3 

Increased Quality 3.4 

Increased Motivation 3.4 

(Not) More Inhibited (R) 3.5 

Better Learning 3 . 6 

Learned More 3 . 7 

Increased Efficiency 3.7 

More Convenient 3 . 7 

Communicated More With Students 3.7 

Stop Participating (R) 4.2 

Less Work 4.9 



Key: Ratings could vary from 1.0 to 7.0. In computing means for 
this display, scoring of negative items was reversed (R) 
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CHAPTER 5 
EFFECTS OF MODE OF DELIVERY 
The purpose of this chapter is to examine differences in the 
Objectively and subjectively measured outcomes of courses, as they 
were affected by mode of delivery. We were concerned with three 
modes of delivery: completely online, mixed, and face-to-faoe. Since 
we have seen that outcomes appear to be strongly related to the 
course, to the school (including its computing environment), and 
perhaps to whether an online course was a fir^t-time or a repeat 
experience for an instructor, it was necessary to use the 
quasi-experimental designs built into this study in order to examine 
the relationship between mode and outcome. Thus, though we will 
include some oneway analyses of variance which simply compare the 
overall means of outcome measures by mode of delivery, the primary 
method of analysis will be a two-way analysis of variance (using the 
SAS General Linear Models procedure) which identifies interactions of 
mode with course, school, or semester (first vs. second offering). 

DIFFERENCES IN SUBJECTIVELY PERCEIVED OUTCOMES, BY MODE 
Of the scores of variables used in this study, very few were 
significantly related to mode of delivery, when all courses delivered 
completely online, in mixed mode, or face-to-face were pooled into 
three groups. Table 5-1 gives the results of most interest. It 
includes the dependent variables based on subjective measures which 
were of primary interest (the indexes) , plus individual items 
measured for all modes which produced statistically significant 
differences. 

There were no significant differences among modes in tl>e overall 
course rating index, interest index, or synthesis index. For the 
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instructor rating index and the collaborative index, the mixed mode 
of delivery was associated with significantly better ratings. 
However, in looking at individual items, it was interesting that the 
mixed mode produced significantly worse ratings in two cases. 
Students in mixed-mode courses reported that the course requirements 
were less clear, and that they were less likely to have completed all 
the written assignments. Apparently, although the mixed mode of 
delivery is exciting and provides very good conditions for 
collaborative learning among students, the combination of traditional 
and online activities can prove overwhelming and confusing for 
students . 

As would be expected, students who used the Virtual Classroom 
were significantly more likely to report increased computer 
competence. Those who had completely online courses were most likely 
to have been stimulated to do additional outside reading related to 
the course. On the other hand, for all courses combined, the 
expectations concerning developing relationships with other students 
online were not bourne out. Students in the totally online courses 
were less likely to report having developed new friendships in the 
class, and less likely to feel that they had developed their ability 
to communicate clearly about the subject. 
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Table 5-1 
COURSE OUTCOMES BY MODE OF DELIVERY 
MEANS AND ANOVA 



VARIABLE 



ONLINE 



MIXED 



F-T-F 



COURSE RATING INDEX 


16. 


0 




15. 


0 




15. 


3 




1.38 


.25 


INSTRUCTOR RATING INDEX 


22. 


1 


A 


19. 


8 


A 


21. 


2 




3.02 


.05 


COLLABORATIVE INDEX 


20. 


6 


A 


24. 


9 


AB 


23. 


0 


B 


20.7 


. 001 


INTEREST INDEX 


10. 


4 




10. 


3 




10. 


0 




.7 


.48 


SYNTHESIS INDEX 


10. 


8 




11. 


3 




11. 


2 




1.7 


. 13 


INCREASED COMPUTER 


2. 


1 


A 


2. 


1 


A 


3. 


1 


AB 


30.95 


.001 



COMPETENCE 

NEW FRIENDSHIPS 2 . 6 AB 2.0 A 

COMPLETED WRITTEN 1.9 A 2 . 2 AB 

ASSIGNMENTS 



STIMULATED ADDITIONAL 
READING 



2.7 AB 3.1 A 



2.2 B 
1.9 B 

3.1 B 



9.44 .001 
4.11 .02 



4.58 



DEVELOPED ABILITY TO 2 . 5 AB 2.1 A 

COMMUNICATE 

COURSE REQUIREMENTS CLEAR 2.1 A 2 . 4 AB 



2.0 B 



4 . 54 



ENTRIES IN THE SAME ROW WITH THE SAME LETTER ARE 
SIGNIFICANTLY DIFFERENT, DUNCAN MULTIPLE RANGE TEST 



.01 



2.3 B 11.24 .001 
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DIFFERENCES IN OBJECTIVELY GRADED PERtORMANCE 
For those courses with matched online and traditional sections, 
one "objective" measure of the influence of mode of delivery on 
course outcomes was the grades obtained. As can be seen in Table 
5-2, there was only one significant difference in grades, when course 
was controlled. However, the picture was very mixed and muddied. The 
number of subjects in each section was small, and thus differences 
would have to be large to be statistically significant. Secondly, 
despite the original plan to give exactly the same midterm, final, 
and assignments in matched sections, and to grade them the same way, 
the .Instructors found that they could not do this. 

In the management course, the instructor reported that the 
assignments completed by students in the section which had the 
Management Lab online, were far superior. However, he felt that he 
should not penalize the students who did not have this facility, so 
he did not grade them on the same standard. 

In the Sociology course, the initial midterm grades on the same 
exam were much worse in the online section. The instructor felt that 
this might have been due to the fact that they had been doing 
assignments that were different than those in the face-to-face 
section, and which were not as closely related to the questions that 
were included in the examination. Therefore, he gave them a chance 
to attend two face-to-face review sessions which did concentrate on 
the types of questions that were on the exam, and to retake the exam. 
Five students availed themselves of this opportunity. The final exam 
in Introductory Sociology was the same and administered under the 
same conditions for both sections, however, and there was no 
difference in scores. The students in the online section did turn In 
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iRoro fltnd btstter written assignments, so their overall course grade 
w«t,s higher, though not significantly so. 

In th« rfiquirv.d freshman level course in statistics at Upsala, 
all grades in all sections tended to be low. It became a matter of 
which failure rates were highest! Perfooianc© was equally poor, on 
th© average, in both sections. 

In the Computer Science course at NJIT, the instructor gave 
additional activities 3.nd asisignments online, because he found that 
the students could complete th-a core material contained in the 
lectures much faster online. For this course, the difference on 
midterm exam scores approaches significance (p** ..12), with the online 
students doing better. Thf-re Wv-as no difference in the final exam 
scores, but when the qi;iality of aii'sigmiicsnts was factored in, the 
instructor judged the online students as having done significantly 
better work, on the average* The online students averaged a solid "B" 
(3.11 on a 4 point scale where A-« 4.00 B*« 3,00, ate. ) » whereas the 
face-to~fac© studc^Jit.'i averaged a C- (3.4.93) >. 

Thus, the overall conclusioi* is that online students learned the 
roqtiired material for a course as well m or better than students in 
face'-to-'face classes. In a cowrse wher© computer usage is intrinsic, 
the performance may tend to be significantly foett«r. At the Freshman 
level, in survey courses in which many students have difficulties 
pa.5sing, even though there m no significant difference in objective 
measures of performance, the instrvictors felt that totally online 
delivery would not be beneficial. The better students did very well 
in thes»2 freshmen IweX cours-ies online, but the weaker students 
tended to drop out or do even more poorly, according the the 
perceptionr^ of the instructors in their course report.s. 



Table 5-2 

DIFFERENCES IN GRADES BY MODE AND COURSE 
QUASI -EXPERIMENTAL DESIGN 



MEAN MIDTERM EXAM GRADE 



COURSE ONLINE 

CIS 213 90.7 

INTRO SOCIOLOGY 75.2 

STATISTICS 68 . 8 

ALL 78.5 



FTF BOTH 

80.1 85.4 

75.9 75.5 

69.5 69.6 
75.2 



F= 2.82 p=.02 
Mode F= .91 p= .34 
Course F= 6.43 p« .003 
Mode by Course F= .98 p= .35 

NO SIGNIFICANT DIFFERENCE? JITHIN COURSES 



MEA. RADE ON FINAL EXAM 



COURSE 



ONLINE 



FTF 



BOTH 



CIS 213 

MATH 305 

INTRO SOCIOLOGY 

STATISTICS 

ALL 



79.3 
79.0 
68 .4 

53.6 
70.1 



78.8 
G1.6 
68.7 
56.4 

71,4 



79.1 
80.3 
68.5 
55. 0 



F= 5.27 p=.001 
Mode F= .13 p« .72 
Course F= 11.28 p« .OO"" 
Mode by Course F= .06 p" i 



NO SIGN'^^ICANT DIFFERENCES WITHIN COURSES 
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FINAL COURSE AVERAGE 



COURSE 


ONLINE 




FTF 




BOTH 


P 


CIS 213 


3 a 11 








A • 




MATH 305 


3.25 




3.16 




3.20 


.85 


SOCIOLOGY 


1.62 




1.47 




1.54 


.74 


STATISTICS 


2.23 




2.35 




2.29 


.78 


MANAGEMENT 


2.68 




2.85 




2.76 


.68 


ALL 


2.53 




2 . 35 










ANOVA 


F= 4 


.27 p= . 


001 








Mode 


F= 1 


.23 p= 


.27 








Course 




7.58 p= 


. 001 






Mode by Course 


F« 1.3 


P= 


.27 
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Measuring changes in writing Scores 
One of the online courses was a freshman writing seminar at 
Upsala. A pre-test of essay writing skill was administered to all 
freshmen before they took this course. During the Spring semester, 
after they completed the course, a similar essay examination was 
given to the students. Both were graded on a holistic basis, as 
follows. The faculty is first "normed" by having all graders 
evaluate some sample essays which are photocopied, and then 
discussing differences in the scores assigned. Two faculty members 
assign a score from 1 to 10 for the essay. These two scores are 
averaged if they are reasonably consistent. If the two scores are 
more than two points apart, a third faculty member scores the exam, 
and then the two most similar scores are used. 

If the students in the section which did assignments online 
improved more than other students as a result, this ought to be 
reflected in a more positive change in their writing scores than 
would be characteristic of students in the totally offline sections. 
However, as can be seen in Table 5-3, this was not the case. In 
fact, their scores went down a fraction of a point. There were no 
significant differences between this section and the traditional 
sections. 

However, this measure also shows no change in holistically 
scored essays for the entire set of courses. In other words, if all 
freshmen in all the writing sections improved their writing in any 
way in a one semester course, this measure did not detect it. 

What happened here? Certainly we have no evidence to conclude, 
on the basis of these scores, that use of the Virtual classroom on 
EIES improved writing. Discussions with the Director of the writing 
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program at Upsala, Jim Stain, provide some possible explanations. The 
holistic grading procedure used at Upsala is neither very sensitive 
to specific types of changes in writing, nor very reliable. The 
graders are all faculty involved in the program and any other faculty 
or administrators who can be recruited to volunteer to grade some 300 
essays during a few hours. Prof. Stam observed that the faculty was 
"hastily normed" and that the scoring does not appear to be very 
reliable. The procedure does show significant change, on the 
average, for the Basic Skills remedial course, which is required of 
all students who score less than 5 on the first exam. (These scores 
do not appear in Table 5-3, since the target course chosen was the 
writing seminar for those deemed not to have serious deficiencies). 
Prof. Stam pointed out that in 14 weeks, each student is usually 
concentrating on improving one or two aspects of their writing. While 
they are concentrating on this aspect, others may actually get worse. 

There was also an interesting methodological problem. All 
students used paper and pens for their pre-test. The traditional 
sections used paper and pens for their writing during the course, and 
the same for the post-test. The students in the experimental section 
used a personal computer, and the text processing built into EIES, 
for all their writing assignments. Then they used paper and pens for 
the post-test writing sample. Perhaps the skills learned for writing 
and revising using a computer and for "talking through your fingers" 
do not carry over to writing in a non- computer-supported mode? 

If we were to conduct an experiment on changes in writing in the 
future, we would change the procedures used here. First of all, 
writing ought to be measured on both the pre- and post-test on a 
number of separate dimensions (e.g., grammar, organization, clarity, 
originality, expressiveness, completeness and length of the essa^ ) . 
O 163-4 r , 
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There should be three conditions: no computer support, micros with 
word processors for students to individually use a computer for 
writing assignments; and the addition of Virtual Classroom for 
exchanging drafts and discussing and commenting on drafts, for some 
sections. Sections which use the computer for writing ought to use 
it for the post-test, since that is how the students will be used to 
writing. 

Instuctors in the non-writing courses were asked if they had 
noticed any changes in their students' writing over the course of the 
semester as they used the system. Most agree that there was 
definitely a tendency for students to write a lot more as the 
semester progressed. 

Paul Levinson, of Connected Education, offers the following 
observations: 

Connect Ed has had one dramatic case of a woman with 
dyslexia or similiar problem. When she first signed up for 
our courses, she was concerned lest her disability prevent 
her from participating. Her first comments were 
intelligent, but short and not very flowing. 

Less than a year later she was uploading 300 line term 
papers that read beautifully. 

Other more common consequences of on-line writing seem 
to be a general increase in the flow and smoothness of the 
writing over a few month period of time. 

Because of the insensitivity and unreliability of the holistic 
scoring methods used, we are not ready to conclude that Virtual 
ClasF.room makes "no difference" in students' writing. A much more 
carefully controlled study would be necessary in order to determine 
what changes in student writing, if any, are more likely or less 
likely to emerge when writing assignments are shared with others 
online, as compared to other modes for teaching writing. 
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Table 5-3 



TEST OF SIGNIFICANT IMPACT ON WRITING SCORES 

F P 
.29 .59 

1.72 .19 

.51 .48 



NOTE; Writing Scores on the two exams may vary from 
a low of one to a high of ten. Anything below 
five is considered to be below minimum college 
level . 





ONLINE 


OTHERS 


TEST 1 MEAN 


6.60 


6.87 


TEST 1 SD 


1 . 45 


.90 


N 


15 


302 


TEST 2 MEAN 


6.29 


6.91 


TEST 2 SD 


1.33 


1.76 


N 


14 


271 


DIFFERENCE MEAN 


-.31 


.04 


DIFFERENCE SD 


1.25 


1.75 


N 


13 


267 
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OUTCOMES BY MODE AND COURSE 
When a two-way analysis of variance was used for all dependent 
variables, employing the matched Fall courses in the 
quasi-experimental factorial design, the results of the previous 
one-way analyses were verified. Almost all differences in outcomes 
were associated with differences among courses, rather than with 
differences among modes. There was some interaction between course 
and mode, but given the small number of cases, interaction was 
generally not statistically significant. With such a small number of 
students in each of the course by mode conditions, differences had to 
be extremely large and consistent to reach statistical significance; 
therefore, even differences significant at only the .10 level were 
worth looking at. 

The five tables which follow present the results for individual 
variables which produced some significant differences, and for the 
indices measuring the dependent variables of primary concern. In 
te^-ms of students reporting that they completed their required 
readings (table 5-4), the primary differences were among courses: 
readings were least likely to be completed in the Management course. 
In this course and in the Sociology course, there was some tendency 
for the readings to be more regularly completed in the face-to-face 
mode, but the difference was not significant. 

For increased interest in the subject matter, once again, there 
was no overall difference by mode, but there was both a difference by 
course and some interaction between course and mode (table 5-5). For 
instance, there was a tendency in both the lower level and the upper 
level statistics co' s and in the Computer Science course for 
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interest to have increased moro in the online sections, but the 
reverse is true in Introductory Sociology. 

Synthesis scores were also apparently affected by an interaction 
between course and mode (table 5-6) . They were higher online in two 
courses, and higher in the traditional sections for three of the 
courses . 

Looking at the overall instructor Rating and Course Rating 
indices (tables 5-7 and 5-8), the earlier findings, that differences 
among courses account for more cf the variance than differences anong 
modes of delivery, are confirmed. For both of these indices, r&uings 
were better for the online sections of Computer Science and the upper 
level statistics course (Math 305) and worse for the online section 
of Introductory Sociology; but once again, few of the individual 
differences withiii course were significant. 

In sum, it was differences among courses that accounted for mos^: 
of the differences in outcome measures. To the extent that there was 
some interaction between mode of delivery and course, the pattern was 
not consistent. Within courses, none of the differences in outcome 
by mode was large enough to be statistically significant, and the 
direction of the differences that occur was mixed. There was a 
fairly consistent tendency for the ratings for Comput?ir Science to be 
higher in the online sections and for the ratings for Sociology to be 
higher for the face-to- face section. 
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Table 5-4 

Completed Required Readings, by Mode and Course 

Means and Anova 



Course 


Online 


MODE 
FTF 


Both 


CIS 213 


2.1 


2.1 


2.1 


MATH :i05 


2.2 


2.6 


2.4 


STATISTICS 


2.2 


2.4 


2.3 


SOCIOLOGY 


2.5 A 


1.8 A 


2.1 


MANAGEMENT 


3.2 A 


2.6 A 


2.9 


All Courses 


2.4 


2.3 





* Conditions wit^i letter A are significantly 
different at 0.05 level 

Anova: F=3.62 p«0.001 

Mode: F=0.77 p=0.382 

Course: F=5.41 p«0.001 

Mode X Course: F-2.19 p=0.072 

Key: 1= strongly Agree 

5« Strongly Disagree 
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Table 5-5 
Interest Index by Mode and Course 
Means and Anova 



Course 


O'.iline 


MODE 
FTF 




Botn 


CIS 213 


11.6 


10.4 




11.0 


MATH 305 


12.0 A 


9.8 


A 


il»0 


STATISTICS 


8.7 


7 3 




8.3 


SOCIOLOGY 


8.7 A 


10.9 


A 


9.8 


MANAGEMENT 


9.3 


10-3 




9.8 


All Courses 


10.0 


9.9 







* Conditions with letter A are significantly 
different at 5 level 

Anova: F«j.74 p»0.001 

Mode: F«0.31 p«0.579 

Course: F«8.09 p«0.001 

Mode X Course: F«5.74 p=0.001 

Ranges 3 (low) to 15 (high) 



Table 5-6 
Synthesis index by Mode and Course 
Means and Anova 



Course 


Online 


MODE 
FTF 


Both 


CIS 213 


11.0 


10.6 


10.8 


MATH 305 


12.3 


11.6 


12.0 


STATISTICS 


9.8 


10.3 


10.0 


SOCIOLOGY 


10.0 A 


11.9 A 


10.9 


MANAGEMENT 


10.7 


11.2 


10.9 


All Courses 


10.8 


11.1 





* Conditions with letter A are significantly 
different at 0.05 level 

Anova: F-2.29 p«0.020 

Mode: F=1.01 p-0.315 

Course: F=3.18 p-0.015 

Mode X Course: F»1.71 p»0.150 

Range= 3 (low) to 15 (high) 
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Table 5-7 

Instructor Rating Index by Mode and Course 

Means and Anova 



Course 


Online 


MODE 
FTF 




Both 


CIS 213 


25.4 


28.5 




27.0 


MATH 305 


14.8 


15.7 




15.2 


STATISTICS 


25.8 


26.9 




26.3 


SOCIOLOGY 


25.5 A 


19.3 


A 


22.4 


MANAGEMENT 


23.0 A 


20.2 


A 


21.6 


All Courses 


22.9 


22.1 







* Conditions with letter A are significantly 
different at 0.05 level 



Anova : 

Mode: 

Course: 

Mode X Course: 



F-12.34 p=0.001 

F= 0.80 p*0.374 

F«20.94 p-0.001 

F= 3.63 p-0.008 



Ranges* n (best) to 55 (worst) 



Table 5-8 

Course Rating Index by Mode and Course 
Means and Anova 



Course 


Online 


MODE 
FTF 




Both 


CIS 213 


14.3 


17.1 




15.7 


MATH 305 


12.5 


13.6 




13.1 


STATISTICS 


19.2 


18.6 




18.9 


SOCIOLOGY 


17.6 A 


13.7 


A 


15.7 


MANAGEMENT 


16.7 A 


14.6 


A 


15.6 


All Courses 


16.1 


15.5 







* Conditions with letter A are significantly 
different at 0.05 level 



Anova : 

Mode: 

Course: 

Mode X Course: 



F«4.42 p^O.OOl 

F«0.62 p=0.431 

F=6.22 p«0.001 

F=2.61 p=0.038 



Range* 7 (best) to 35 (worst) 
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INTERACTIONS OF MODE AND SCHOOL 

"School," as w® have previously noted, v/as reiUted to 
differences in Virtual Cls^ssrooa outcomes not only bec^tuse of 
differencess in equipment access conditions, but also becaus© it was 
confounded with diffv®r©nc©s in the level of the courses which v/ere 
offered. At NJIT, the online courses were for undergraduate students 
in the Sophomore to Senior years? at Upsala,, the totallY online 
courses were Freshman-l«vel, v/hile the misced-Tnode courses were for 
upper level undergraduate courses; and for Connected Education and 
OISE, the courses were at the post-graduate level, and all students 
had their own raicrocoroputers * Thus, it is not surprisincy that there 
was an interaction between "school" and mode for most of the outcome 
variables. Included here are only the wost important of the results 
of these analyses; most outcome variables showed results chat varied 
simultaneously by school as well as by icaode of delivery. 

In the first table (5-9) in this series of selected significant 
interactions by reode and school, we see that the students' 
perceptions of problems with sufficient access to a tenninal or 
microcomputer are in some ways different than laight have been 
imagined. For the remote education students in Connected Education 
and OISE, as would be expected, acctfesa was not a problem. However, 
the surprising things were that student perceptions of access 
problems were higher at NJXT than we assumed they wo\ii.Ji be, and at 
Upsala, for unclear reason*}, t\m access problems were considered more 
serious in the mixed-mode course© than in. the totally online courses. 
This may be because those in the totally online course© were prepared 
to have to go to the microlab to use computers, while those in the 
mixed mode courses had not chosen that mode and resented the trip 
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The next three tables show the results for some specific course 
outcoFies. In terms of dlev€*loping an increased ability to communicate 
one's ideas clearly, the best outcomes were for the mixed mode 
courses at Upsala (Table S-IO) ♦ For improving one's ability to 
critically analyse written material, the students in the totally 
online courses at NJIT reported significantly higher levels of 
improvement, while those in the mixed modes courses at Upsala were 
most likely to perceive improvements in this area (Table 5-11) . For 
increasing confidence in expressing one's ideas (Table 5-12), the 
pattern of significantly better results in Upsala mixed modes courses 
than in either totally online or totally face-to-face courses 
continued. At NJIT, the mixed modes condition also resulted in the 
best overall ratings on this outcome criterion. 

The next set of results turned to some overall outcome ' idices 
that applied to results for all three modes. The best overall scores 
on the "Increased interest" index (Table 5-13) were for the remote 
education students, the NJIT totally online courses, and the Upsala 
mixed modes courses. For degree of collaborative learning, the index 
scores were highest for the mixed-modes condition, at both NJIT and 
Upsala. Instructor rating indexes tended to be highest for totally 
online courses at NJIT, and for the mixed-mode courses at Upsala 
(Table 5-14) . 

The final table in this series is for oitcomes measured only for 

those students who used Virtual Classroom, and who compared it to 

previous face-to-face courses. For the VC Overall rating index 

(Table 5-15), the best ratii^gs ocoured for the mi:'ed modes delivery 

at NJIT and the totally online remote education students. At both 

NJIT and Upsala, the mixed modes students gave higher overall ratings 
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to the Virtual classroom than did the totally online students, though 
neither difference was statistically significant. 
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Table 5-9 

Terminal Access Problem, by Mode and School 

Means and Anova 



MODE 



School 


Online 


Mixed 


Both 


NJIT 


3.5 


3,9 


3.6 


UPSALA 


3.8 A 


2.9 A 




CONNECT-ED 


4.8 






Others 


4.6 






All Schools 


4.2 







* Conditions with letter A are significantly 
different at 0.05 level 

Anova: F=4.08 p=0.001 

Mode: F«0.74 p=0.478 

School: F=4.27 p=0.006 

Mode X School: F=8.30 p=0.004 

KEY: 1= Serious Problem 5= Not a problem 
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Table 5-10 



Developed Ability to Communicate Clearly 
by Mode and School 

Means and Anova 



MODE 



School 


Online 


Mixed 


FTP 


All 


NJIT 


2.3 


2.2 


2.2 


2.2 


UPSALA 


2.9 AB 


2.0 AC 


2.4 BC 


2.4 


CONNECT-ED 


2.9 








Others 


2.4 








All Schools 


2.6 









* Conditions with letter A,B, & C are significantly 
different at 0.05 level 

Anova: F=3.30 p=0.001 

Mode: F»5.94 p»0.003 

School: F«1.82 p«0.144 

Mode X School: F=s2.88 p«0.036 

Key: l» Strongly Agree 

5= Strongly Disagree 
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Table 5-11 

Improved Critical Analysis Ability, 
by Mode and School 

Means and Anova 









MODE 




School 


Online 


Mixed 


FTP 


All 


NJIT 


2.2 A 


2.6 A 


2.3 


2.4 


UPSALA 


2.6 


2.3 


2.6 


2.5 



CONNECT-ED 3 . 6 
Others 2 . 7 

All Schools 2.8 



* Conditions with letter A are significantly 
different at 0.05 level 

Anova: F«3.44 p=0.001 

Mode: F=0.13 p«0.881 

School: F=4.97 p=0. 002 

Mode X School: F=2.66 p=0.019 

KEY: 1= Strongly Agree 

5= Strongly Disagree 
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Table 5-12 



Increased Confidence in Expressing ::deas, 
by Mode and School 

Means and Anova 



MODE 



School 


Online 


Mixed 


FTF 


All 


NJIT 


2.2 


2.0 


2.2 


2.1 


UPSALA 


2.7 A 


2.2 A 


2.5 


2.4 


CONNECT-ED 


2.7 








Others 


2.4 








All Schools 


2.5 









* Conditions with letter A are significantly 
different at 0.05 level 

Anova: F=2.91 p^O.OO-^ 

Mode: F-5.41 p=0.005 

School: F=4.09 p=0.007 

Mode X School: v=1.67 p«0.174 

KEY: 1= Strongly Agree 5= Strongly Disagree 
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Table 5-13 
Interest Index by Mode and School 
Means and Anova 



MODE 

School Online Mixed FTF All 

NJIT 11.0 A 9.9 A 10.2 10.4 

UPSALA 8.9A 10. 6A 9.6 9.7 

CONNECT-ED 11.2 

Others 11.4 

All Schools 10.6 



* Conditions with letter A are significantly 
different at 0.05 level 

Anova: F=3.59 p=0.001 

Mode: F=0.60 p=0.550 

School: F=3.54 p=0.015 

Mode X School: F=5.02 p«0.002 

KEY: index range= 3 (low) to 15 (high) 
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Table 5-14 

tructor Rating Index by Mode and School 
Means and Anova 



MODE 



School 


Online 


Mixed 


FTF 


All 


NJIT 


19.4 


20,4 


21.0 


20.3 


UPSALA 


25.9 A 


19. C >B 


22.9 B 


22.6 


CONNECT-ED 


27.0 






Others 


19.9 








All Schools 


23.0 









* Conditions with letter A & B are significantly 
different at 0.05 level 

Anova: F»4.20 p«0.001 

Mode: F=3.80 p«0.024 

School: F=4.17 p«0.007 

Mode X School: F=4.70 p=0.003 

Key: Index range= 11 (best) to 5b (worst) 
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Table 5-15 
VC Overall Index by Mode and School 
Means and Anova 



MODE 



School 


Online 


Mixed 


Both 


NJIT 


19.1 


20.9 


20.0 


UPSALA 


14.2 


15.6 


14.9 


CONNECT-ED 


17.0 






Others 


21.4 






All Schools 


17.9 






Anova : 




F« 6.62 


p=0.001 


Mode: 




F= 2.49 


p«0.117 


School: 




F=10.28 


p=0.001 


Mode X School 


• 
* 


F=: 0.03 


p=0.854 


Key: Index range= 4 


(lowest rating) to 28 
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EFFECTS OF REPEATING COURSES A SECOND TIME 

Four of the courses which were totally or partially online the 
first semester were repeated the second semester. The assumption was 
that with experience, not only would the process of teaching online 
be easier for the instructor, but it would also result in better 
outcomes perceived by the students. 

There was a tendency for courses to improve the second time they 
were offered online, but there are many exceptions to this 
generalization when specific courses and outcomes are examined. 
Taking the overall results first, outcomes for the overall student 
rating index for the virtual Classroom are shown in Table 5-16. It 
was true that these overall ratings were better che second semv^ster 
for all courses that were repeated. However, only the Management Lab 
showed a statistically significant improvement. 

In terms of final grades assigned to students, which measured 
the instructor's perceptions of the students' performance, there were 
no significant differences (Table 5-17). Perhaps this was to be 
expected, since instructors may tend to grade on a curve for any 
class. There was also a mix in the direction of the non-significant 
differences in average grades that did occur: grades were higher the 
first semester in the Upsala statistics course and CIS 213 at NJIT, 
and higher the second semester for the Management course. 

The management course was the only one which tended to 
consistently show significant improvement the second semester on one 
outcome measure after another. Looking at interest in the subject 
matter, for instance, this was the only difference between semesters 
which was significant (Table 5-18). The same was true for increases 
in the perception of Collaborative Learning (Table 5-19) . Looking at 
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the scores on the instructor rating index (Table 5-20) , the second 
semester the instructor was rated significantly better only for CIS 
213 and the Management course. For the Math 305 course, the 
instructor rating was actually better the first semester; however, 
since this instructor had exceptionally high ratings in all modes and 
semesters, we may be seeing a kind of "regression effect." 
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VC Overall Rating Index by Semester and Course 

Means and Anova 



SEMESTER 



Course 


1 


2 


Both 


CIS 213 


19.4 


20.5 


20.0 


MATH 305 


17.0 


19.7 


18.3 


STATISTICS 


13.9 


14.3 


14.1 


MANAGEMENT 


18.8 A 


23.0 A 


20.9 


All Courses 


17.3 


19.4 





* Conditions with letter A are significantl 
different at 0.05 level 



Anova : 
Course : 



einester ! 



Course X Semester: 



F=4.10 p«0.001 

F«6.86 p=0.001 

F»3.31 p-0.072 

F=0.7C p=»0.556 



Key: Index may range from 4 (lowest) to 28 
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Table 5-17 
Final Grade by Semester and Course 
Means and Anova 







SEMESTER 






1 


2 


Botn 


ClH 213 


3.1 


2.8 


2.9 


MATH 305 


3.2 


3,3 


3.2 






1 A 




MANAGEMENT 


2.7 


3,1 


2.9 


All Courses 


2.7 


2.6 




Anova : 




F= 5.70 


p=0.001 


Course: 




F«ll,95 


p=0, 001 


Semester: 




F= 0.03 


p«0.865 


Course x Seme 


ster: 


F= 0.79 


p~0. 505 


Key: A^ 4.0, 


B= 3.0 


etc. 
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Table 5-18 
Interest Index by Semester and Course 
Means and Anova 

SEMESTER 



Course 


1 


' 2 


Both 


CIS 213 


11.6 


10.9 


11.2 


MATH 305 


12.0 


10.8 


11.4 


STATISTICS 


8.9 


9.0 


9.0 


MANAGEMENT 


9.3 A 


10.5 A 


9.9 


All Courses 


10.4 


10.3 





* Conditions with letter A are significantly 
different at 0.05 level 



Anova : 

Course: 

Semester: 

Course X Semester: 



F=3.92 p=0.001 

F=7.56 p=0.001 

F=0.15 p=0.704 

F=2.27 p=0.084 



KEY: Index scores may range from 3 (low) to 15 
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Table 5-19 

Collaborative Index by Semester and Course 

Means and Anova 



SEMESTER 



Course 


1 


2 


Both 


CIS 213 


20.0 


18.8 


19.4 


MATH 305 


22.1 


21.7 


21.9 


STATISTICS 


21.0 


19.4 


20.2 


MANAGEMENT 


24.7 A 


27,2 A 


25.9 


All Courses 


22.0 


21.8 





* Conditions with letter A are significantly 
different at 0.05 level 

Anova: F= 8.10 p=0.001 

Course: F=16.58 p=0.001 

Semester: F« 0.05 p=s0.815 

Course X Semester: F= 1.76 p='0.159 

Key: Collaborative Index may range from 6 (least) 
to 34 
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Table 5-20 

Instructor Rating Index by Semester and Course 

Means and Anova 



SEMESTER 



Course? 


1 




2 






Both 


CIS 213 


25.4 


A 


20. 


5 


A 


22.9 


MATH 305 


14.8 


A 


19. 


2 


A 


17.0 


STATISTICS 


25.8 




25. 


9 




25.8 


MANAGEMENT 


23.0 


A 


18. 


0 


A 


20.5 


All Courses 


22.2 




20. 


9 







* Conditions with letter A are significantly 
different at 0.05 level 

Anova: F= 7.25 p=0.001 

Course: F«12.17 p=0,00i 

Semester: F» 1.54 p-»0.217 

Course X Semesters F= 5.29 p~0.002 

Key: Index may range from 11 (best) to 55 
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SUMMARY 

The previous chapter examined the results of subjective 
assessments of the Virtual Classroom by students who had experienced 
either partially or totally online courses, and who were asked to 
compare it with their previous experiences in face-to-face courses. 
The students reported VC to be different in many ways, including more 
convenience, better access to the professor, more involvment, but 
also more work. 

This chapter analyzed differences among modes of delivery by 
using data from a quasi-experimental design. Different students wtre 
given different courses in different modes, but asked the same 
questions (and within course, given the same examinations) . The 
reasoning was that if mode of delivery was a strong causal factor in 
influencing outcomes, this should show up as significant differences 
in the responses of the students receiving different "treatments." 

Our samples of students within each mode and course condition 
were too small to provide much statistical power, but generally 
speaking, there were few variations in outcome associated with mode 
of delivery. There were constantly large and significant 
differences among the courses and among the schools. 

In terms of grades, the only statistically significant 
difference was for the Computer Science course, where grades were 
better in the online section. This was also the course for which 
students in the Virtual Classroom condition spent the most time 
online. 

An attempt to determine whether the use of VC might help improve 
progress ivi a freshman level writing course was a failure. 
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Holistically graded pre-and post-course essays showed no change in 
scores l!or the VC section, but also showed nc change for all of the 
other sections. Thus we cannot determine whether the medium has no 
effect, or the results are due to an unreliable and insensitive 
scoring procedure. 

When looked at by mode and school, the poorest results occured 
for the totally online, freshman-level courses at Upsala. The 
upper-level, mixed modes courses at Upsala tended to be rated 
relatively well; for instance, these courses had relatively high 
ratings for items on developing ability to communicate clearly, to 
improve critical analysis ability, increased confidence in expressing 
ideas, and increased interest in the subject matter. Thus 
significantly different outcomes by school and mode may be partially 
an artifact of differences in the level of maturity of the students 
enrolled in totally online courses in the two schools. The 
mixed-modes courses at Upsala were all upper-level; students in 
upper-level courses tend to be more mature and more consistently 
"ready" for an intensive college-level learning experience than is 
average student in the freshman-level courses that were totally 
online at Upsala. 

There was a tendency for student ratings of courses to improve 
the second time they were offered online, but there were many 
exceptions to t^is generalization, when specific courses and outcomes 
were examined. For instance, although the overall ratings of the 
Virtual Classroom experience were higher the second time for all four 
courses that were repeated, only the ratings for the Management 
course shov/ed a statistically significant improvement for that index. 
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CHAPTER 6 

STUDENT ATTRIBUTES AND BEHAVIOR RELATED TO OUTCOMES 
We have seen in Chapter 5 that some of the differences in 
outcomes of either totally online or mixed-mode courses are 
associated with the context provided by the course, the school and 
the access conditions available there, and whether a course is a 
first-time or a repeat offering. In this chapter, we will see that 
there were also many significant differences associated with student 
attitudes, attributes, and behavior. In the analyses summarized 
here, students in traditional courses were eliminated, and those in 
the partially and totally online sections were grouped together. 

Student Characteristics as Predictors 

Pre-Use Expectations become Self-Fulf illing Prophecies 

Table 6-1 displays the correlations between pre-use variables 
and course outcomes. As would be expected, those with more positive 
attitudes towards computers at the outset were more likely to report 
more favorable course outcomes, to spend more time online, and to log 
on more frequently. They were also more likely to report that EIEG 
was "easy to learn," less likely to feel at ::he end that they would 
not choose to take another online course, and rated the Virtual 
Classroom mode of delivery more favorably in comparison to 
face-to-face clas 

These same correlations tended to repeat and to be stronger when 
pre-use expectations about the EIES system in particular, rather than 
general attitudes toward computers, were used as the predictor. The 
implication is that participation in the Virtual Classroom mode of 
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learning should ideally be a ch- re. - 3 student, so that those 
with poor initial attitudes arv x ..ced to take part. Several of 
the interviews in the Appendix examples of the "most negative" 

of the students who participated support this interpretation of the 
correlations. For instance, in interview 9, the student mentioned a 
"lot of apprehension" at the beginning, followed by only once a week 
participation, in this and other cases of negative attitudes and 
inadvertent enrollment, there was a problem with effectively 
communicating with such students to "counsel them out." They seemed 
not to hear what they were told or to read or understand printed 
material directed at them. For instance, the interview 9 student 
complained about NJIT facilities not being open during the weekends; 
yet, both at training and in follow-up announcements, all students 
were informed of the special laboratory where Virtual Classroom 
students could receive assistance. This lab was open half -days on 
Saturdays, and unattended terminals were available all day on 
Saturdays . 

Similarly, in interview 2, with a negative Math 3 05 student, the 
student complained that the fact that the course would be online was 
a total surprise to him, and that he didn't like that idea from the 
beginning. He claimed that it wasn't in the registration material 
(then admits, "Maybe it was, but I just missed it.") OFFERED VIA 
COMPUTER was prominently printed in all-capital letters next to the 
course name and section number for online courses, in the 
registration material, and posters and flyers were placed around the 
registration area. Then there was the telling little detail in 
interview 7 with a dropout, who carefully spelled out the 
instructor's name— getting both the first and last names wrong. 

It is probably not coincidental that all three of these students 
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who started out with being "surprised" to learn about the online 

class at the first meeting, and with negative attitudes toward the 

experiment, work full time and normally were on campus only to 

attend class. They understandably felt overloaded and were likely to 

screen out anything that did not seem to "require" their attention. 

Tho interview 2 student stated, for instance, 

I don't have enough time in my day as it is... I usually go to 
work, then to school, then to work and then back to the house to 
study at 11 at night, and I didn't want to sit down and read 
some other stuff... To sit down and make myself do something 
like that... I don't have the self discipline for it. 

Sphere of Control; Not a Good Predictor 

Qualitative observations similar to those above led initially to 
the inclusion of the Sphere of Control indices as predictors, it was 
hypothesized that considerable self -discipline and ability to manage 
one's time and one's life would be necessary in order to participate 
regularly and sucessfully in a "sign-on anytime" Virtual Classroom 
experience, and the Sphere of Control measures were assumed to tap 
this dimension. However, the results for Sphere of Control indices 
were not as strong or consistent as was hypothesized. The Personal 
Eii. jacy Sphero of Control index was significantly related to the 
overall course outcomes index, and to the perception that EIES was 
easy to learn. Interpersonal Sphere of Control was significantly 
related to the Instructor Rating Index, and to disagreement with the 
statement that they would not choose to take another online course. 
However, neither sphere of Control index was related to the overall 
rating of the virtual Classroom and even those correlations which 
were significant were not very strong. 
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student Maturity and Ab ility are crucial 

"Class standing" corresponded to the educational level of the 
student: freshman through graduate student. Thus, it reflected both 
age and previous academic experience, and could be an indirect 
measure of cognitive maturity. The higher the academic l«vel of the 
student, the less likely they were to conclude that they would not 
take another online course, and the better their overall rating of 
their Virtual Classroom experience in comparison to previous 
face-to-face courses. 

Since many of the students were freshmen, we were missing many 
Grade Point averages, so Math and Verbal Scholastic Aptitude test 
scores were used to explore the relationship between academic ability 
and achievement (whatever combination of these were measured by the 
SAT's) , and process and outcomes in the Virtual classroom 
environment, selected results are displayed in Table 6-2. Many of 
these correlations were moderately strong, and very interesting. 

On the whole, it was the Mathematics SAT score which predicted 
student success in the Virtual Classroom, much more than the Verbal 
SAT score. The first two correlations in Table 6-2 were included as 
a matter of general interest: high Sphere of Control indices were 
associated with high Verbal SAT's but not significantly associated 
with Math SAT scores. Those with high Math SAT's (but not those with 
high Verbal SAT's) signed on significantly more frequently, and also 
spent more total time online and sent more private messages. They 
were less likely to feel inhibited online; more likely to feel that 
they were more involved in the VC course than in traditional courses. 
The high Math SAT students also earned significantly higher final 




course grades online, were more likely to rate course outcomes 
highly, and were much more likely to give the Virtual Classroom 
better ratings overall than the traditional classroom. 

By contrast, many of the correlations for the Verbal SAT are 
either weak (e.g., the weak but insigificant correlation with course 
grade) , OR ACTUALLY REVERSED. This is very intriguing and vas not 
expected. The high Verbal SAT students were significantly less 
likely to feel that VC increased access to the professor or their 
active involvement in th^a course. One can speculate about the 
combination of high Math SAT/Low Verbal SAT as one for which students 
are especially likely to "bloom" in the VC environment, but until we 
combine several year's samples and have a larger number of cases to 
work with, this will have to remain speculation. 

In terms of the association between other student 
characteristics measured and the outcomes, the results tended to be 
mixed and weak, and were not included in tables here. For gender, 
the males did slightly better on final course grades (point biserial 
R= .13, p« .05). Males were also slightly more favorable, on the 
average, towards overall assessment of the Virtual Classroom (R* 
-.16, p« .02). This seems to be related to the tendency for males to 
like computers better and to have higher Math SAT's. The correlation 
between gender and post-course computer attitudes was of a similar 
magnitude: R« -.18 (with females coded as "2"), p« .01. However, 
though statistically significant, the differences related to gender 
were so slight as to have no practical importance. In fact, if one 
wanted to take the "long view," giving females a computer-intensive 
experience in a VC course could be seen as one way to improve their 
computer-related skills and attitudes. 

The only correlation of outcomes with nationality was a slight 
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(R=:.17 p« .03) tendency for non-Americans to feel that they were less 
able to improve their ability to pull together or synthesize the 
variety of materials presented in courses. In terms of native 
language, the only statistically significant difference was that 
those whose native language was not English were slightly less likely 
to report increased interest in the subject matter (R= .18, p= .01). 

There was only one sta*-.istically significant correlation with 
typing ability at pre-use. Those with better typing skills had 
slightly better attitudes toward computers as measured post-course 
(R= .17, p= .02) . 
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Table 6-1 

Pearson's Correlation Coefficients Between 
Student Characteristics and Selected Outcome Measures 





Computer 
Attitudes 


EIES 
Expec- 
tations 


Personal 
soc 


Inter- 
Personal 


Class 
Standing 


Course Outcome 
Index 


-.12 


-.19 


-.16 


-.08 


-.10 


P 


.04 


.01 


.01 


.11 


.07 


Instructor Rating 
Index 


-.02 


-.06 


-.10 


-.13 


-.04 


P 


.40 


.25 


. 06 


.03 


.27 


VC Overall Index 
P 


.34 
.001 


.38 
.001 


.07 
.20 


.0/ 
.20 


.16 
. 02 


EIES EASY TO LEARN 
P 


.43 
.001 


.40 
.001 


.24 
.002 


.22 
.10 


.14 
. 05 


Not take another 
p 


.31 

. 001 


. 33 


.10 

1 1 
• XX 


.16 

. U2 


.25 
. 001 


Total Hours On 
P 


.15 
.02 


.25 
. 001 


.03 
.34 


-.01 
.43 


09 
.08 


Total Tiroes On 
P 


.21 
.001 


.26 
.001 


.11 
.07 


.01 
.44 


.14 
.02 
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Correlations between 


Table 6-2 
SAT Scores and 


VC Process 


?.nd Outcome 


Variable 




SAT MATH 


SAT VERBAL 


Personal SOC 
(N) 

P 




.18 
103 
.94 


.29 

103 
. 002 


Interpersonal SOC 

P 




.15 
.06 


.29 
. 001 


Total Times On 
P 




. 39 
.001 


. 04 
. 34 


(Not) Inhibited 






. 13 


P 




.02 


. 10 


Access Professor 




-.06 


.20 


P 




. 26 


. 02 


More Involved 
P 




-.15 
.08 


. 17 
. 05 


Final Grade 
P 




.31 
.001 


.13 
. 10 


Course Outcome Index 

p 






01 

A A 

. 44 


VC Overall Index 




.36 


.04 


P 




.001 


.35 



O I 1 

199 



Access Conditions, Activity Patterns , and Outcomes 
The first three columns in table 6-3 deal with aspects of 
"access" to the Virtual Classroom: having a micro at home, perceived 
problems with equipment access, and overall "convenience" of the vc 
mode. There were fewer and weaker correlations between having one's 
own microcomputer at home, amount of use of the system and reactions 
to it, than might be supposed. Though the correlation with overall 
vc rating was statistically significant, it was only .18. A second 
measure of access was a question asked on the post-course 
questionnaire about access to a terminal being a serious problem. 
Those who felt it was not a problem were more likely to feel that vc 
had increased the quality of their education, and to give more 
positive overall reactions to the Virtual Classroom mode. 

However, access is more than merely problems getting a terminal 
or micro to use. It may include perceived problems with telephone 
lines; or perhaps, perceived problems in making time to participate. 
The relationship between the question rating whether or not the 
overall convenience of using the VC mode was greater or less than the 
convenience of the traditional classroom was a stronger predictor 
than the items specifically focussed on equipment. The "convenience" 
question was significantly related to the final exam grade and final 
course grade, as well as to subjective ratings of extent of 
collaborative learning, increased interest in the subject, increased 
ability to synthesize material in the field, attitudes toward 
computers at the end of the course, rating of the instruc^.or and the 
course, and in particular, overall rating of VC. 

All of the measures of amount of use of the Virtual classroom 
tended to be related to outcome measures; the number of sessions or 
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total number of times a student signed online was most sstrongly 
related. For instance,, the correlation between number of sessions 
and the final exam score was .34, which was moderately strong. Level 
of activity was also related to the final course grade, a perception 
that VC increased the quality of education, more positive post-course 
attitudes toward computers, and the overall course rating index. 

As in the pilot studies, there were strong and consistent 
relationships between perceptions of having communicated more with 
the professor and the other students online, and overall evaluations 
of the Virtual Classroom experience (table 6-4). Those who felt they 
had better access to their professor, and who read and valued the 
comments and assignments of other students, felt that the Virtual 
Classroom was a better mode of learning than traditional face-to-face 
classes. Those who did not actively take advantage of the 
communication opportunities for such a collaborative style of 
learning tended to prefer the face-to-face mode. 

This is reinforced in the inter\^iews with very positive and very 
negative students in the Appendix. There were two major 
detenrdnants, thus, of outcomes of the Virtual Classroom experience. 
One was whether the students had the self -discipline to regularly 
sign online. The other was whether they used the system to interact 
with the ideas and suggestions of the other students as well as their 
instructor. These two aspects of online behavior were inter-related. 
For thosfi who valued communication with other members of the class, 
motivation to sign online frequently was increased. Frequent, 
regular, and active participation helped them to do well in the 
online course, and contributed to their positive evaluations of the 
course, the instructor, the attainment of learning goals, and 
evaluations of this mode of educational delivery. 
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Table 6-3 



Access and Activity Conditions, by Outcomes 
Pearson Correlation Coefficients 
(N of cases* 163) 







ACCESS 






ACTIVITY 






HOME 


ACCTERM 


CONVEN 


TTOT 


ONTOT 


PRTOT 


FINAL GRADE 
P 


.06 

.23 


. 10 
. 12 


. 33 
.001 


. 16 
. 02 


.22 
.001 


.17 
. 02 


FINAL EXAM 
P 


.06 

.33 


.01 
.48 


-.30 
.02 


.25 
.05 


.34 
.01 


.28 
. 03 


COLLABORATIVE 
INDEX 


.02 


.02 


-.15 


. 14 


.07 


.01 


P 


.39 


.41 


. 03 


. 05 


.19 


,45 


INTEREST INDEX 
P 


.02 
.40 


. 14 
.20 


-.33 
.001 


. 12 
.06 


.17 
.01 


.08 
. 14 


SYNTHESIS INDEX 

n 
f 


-.12 


-.02 


-.26 
• 001 


.08 
. 14 


.07 

. 17 


.03 
.35 


INCREASED 
QUALITY 


.07 


.31 


-.51 


. 16 


.17 


.14 


P 


.18 


.001 


.001 


. 02 


.01 


.04 


COMPUTER ATTS2 
P 


.30 
.001 


. 37 
.001 


-.53 
.001 


. 26 
. 001 


.31 

001 

a W \J Am 


.31 

m \J\J X 


INSTRUCTOR 
RATING 


-.05 


-.12 


. 32 


-.08 


-.11 


-.13 


P 


.23 


.07 


.001 


.16 


.09 


.05 


COURSE RATING 
P 


.06 
.23 


-. 14 
.03 


.38 
.001 


-.20 
.01 


-.20 
.01 


-.13 
.05 


VC OVERALL 
P 


. 18 
.01 


. 36 
.001 


-.63 
.001 


.22 
.001 


.25 
. 001 


.22 
. 001 



KEYS; HOME* Have a terminal at home, pre-use 

ACCTERM* Post question on problems with terminal access 

CONVEN* Agreement with statement that VC is more convenient 

TTOT« Total time online during course 

ONTOT= Number of sessions online during course 

PRTOT« Number of private messages sent during course 
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Table 6-4 



Process and Assessmcints of the Virtual Classroojn 



COMMUN- 
ICATED 


ACCESS 

PROF 


INCREASE 
MOTIVE 


INVOLVED 


COMMENTS 


ASSIGNS 


FINAL GRADE 
P 


-".15 
.04 


• JL / 

. 02 


- . 23 
.001 


-. 22 

.001 


-.11 
.09 


-.11 
.10 


FINAL EXAM 
P 


-.09 
.04 


-.28 
.02 


-.23 
.001 


-.23 

.001 


-. 11 

.09 


-.06 
. 34 


COLLABORATIVE 
INDEX 


-.51 


-.35 


-.25 


-.40 


-.45 


-.30 


P 


*001 


.001 


.001 


.001 


.001 


.001 


INTEREST INDEX 
P 


-.25 
.001 


-.41 
.001 


- . 40 
.001 


-.38 
.001 


-.40 
.001 


-.39 
. 001 


SYNTHESIS INDEX 
P 


-.32 
.001 


-.44 

.001 


-.43 
.001 


-.37 
.001 


-.34 
.001 


-.33 
. 001 


INCREASED 
QUALITY 


-.31 


--,46 


-.36 


-.45 


-.35 


-.35 


P 


.001 


.001 


.001 


.001 


.001 


. 001 


COMPUTER ATTS2 
P 


-.24 
.001 


-.35 
.001 


-.39 
.001 


-.42 
.001 


-.31 
.001 


-.39 
.001 


INSTRUCTOR 
RATING 


.27 


. 35 


.32 


.28 


.21 


.23 


P 


.001 


.001 


.001 


.001 


.001 


.001 


COURSE RATING 
P 


.29 
.001 


.40 
.001 


.46 
.001 


.46 
.001 


.33 
.001 


.32 
.001 


VC OVERALT 
P 


-.41 

.001 


-.60 
.001 


-.48 
.001 


-.64 
.001 


-.44 
.001 


-.48 
.001 



KEYS: 

COMMUNICATED^ Communicated more with other students 
ACCESS PROF« Provided better access to the proffesor 
INCREASE MOTIVE- Fact that assignments would be read by other 

students increased motivation 
INVOLVED* Felt more involved in taking an active part 
COMMENTS* Found comments made by other students useful 
ASSIGNS* Found reading assignments of other students useful 
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Multivariate Analyses 
In various parts of this report, we have noted a series of 
bivariate relationships and relationships which took into account the 
interaction of two variables at a time. What happens when we put all 
our predictors together? Which ones make the biggest contribution to 
explaining the variance in the dependent variables, and which ones 
are not significant once the others are taken into account? 

Because our sample size was fairly small, we did not conduct 
many multivariate analyses or try to push the variance accounted for 
too far. The problem is that as you add variables with a small 
sample, you run out of degrees of freedom; for example, nine 
variables will always explain the variance in ten cases perfectly. 

We used simultaneous regression, which takes all the variables 
in the equations into account at the same time. This does have the 
methodological weakness that if two variables are strongly 
associated, then they will probably share variance accounted for 
between them, and neither one may end up statisically significant. 
However, without a prior theory which clearly predicted what 
variables would be the strongest causes, there was no basis for 
alternative regression procedures, in order to use "mode" and 
"course" as variables, a series of "dummy variables" were constructed 
with 0-1 values (e.g., in the dummy variable for the statistics 
course, it was coded as "1" and all other courses were coded "o," or 
"not statistics.") 

In the first equation (Table 6-5) , all students in all modes at 
NJIT and Upsala were considered, and the dependent variable was the 
course Rating Index, in interpreting the signs of the beta 
coefficients, which are the best overall comparative measure of the 
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level of association with the dependent variable, one must be aware 
of how the variables were coded, which is shown in the questionnaire 
items in the Appendix. The course rating scale was first introduced 
in Chapter 2 on methodology. Because it consisted of a series of 
positive statements accompanied by Likert-type scales which were 
displayed and scored as "1=* Strongly Agree," the lower the total 
score, the more positive the total course rating. 

The strongest predictors have nothing to do with mode of 
delivery. The required Freshman-level statistics course at Upsala 
received the lowest course ratings. Another course taken by many 
freshmen to fulfill a requirement. Sociology, showed up as also 
significantly associated with relatively poor course ratine,^. Only 
two schools were used in this analysis, with NJIT coded "1" and 
Upsala coded "2." The second strongest predictor of course ratings 
was school; despite the two specific courses with relatively low 
ratings, course ratings on the whole were better at Upsala. The 
third strongest predictor was a measure of general ability; students 
with high Math SAT scores rated their courses significantly better. 

Mode of delivery does appear as making a significant 
contribution to predicting overall course ratings: the mixed mode 
courses have lower ratings than the other modes, when everything else 
was simultaneously taken into account. Since on the majority of 
measures, mixed mode courses fared well, we will not make a great 
deal of its appearance in this particular equation. 

The second and third equations are only for those students who 
had a part;ially or totally online course, since it uses variables 
available only for these students. The only two significant 
contributors to predicting final grade in these courses (Table 6-6) 
are SAT verbal score and agreement that taking online courses is more 
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convenient. However, it should be noted th<,t even with twelve 
predictors in the equation, we cannot accurately predict fin^l course 
grades, with only 14% of the variance explained. 

The most important equation for our purposes is the prediction 
of overall rating of the Virtual Classroom (Table 6-7). The total 
proportion of variance explained by the 18 predictor variables is a 
respectable 67%. The significant predictors are SAT Math scores, and 
perceptions that the Virtual Classroom is more convenient than the 
traditional classroom, that it increased access to the professor, and 
that the student was more involvcid in taking an active part in the 
course . 

In a stepwise multiple regression approach to predicting overall 
VC ratings (not included here) , the order of selection was feeling 
more involved in the course, feeling that the VC is more convenient, 
perception of better access to the professor, and the SAT Math score. 
These four variables accounted for 60% of the variance (adjusted R 
squared) . 
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SUMMARY: PREDICTING STUDENT REACTIONS TO THE VIRTUAL CLASSROOM 
"Course" is a much stronger predictor of differences in course 
outcomes than is mode of delivery, Bound up with course are 
differences in characteristics of the students enrolled, in the 
subject matter and thus content of the experiences, and especially, 
differences in teacher style or skill in various modes. 

Our primary interest in this chapter was in pursuing the 
question of correlates of relatively "good" outcomes in Virtual 
Classroom courses. Some student characteristics, such as Math SAT 
scores, are strong predictors of relatively good outcomes. 
Convenience of access is also very important, as is regular and 
active participation, and a perception of improved access to the 
professor. These latter two variables, while partially related to 
student characteristics such as self -discipline, could also be 
greatly affected by how the instructor conducts the online course. 
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TABLE 6-5 



Predicting Course Rating: Multiple Regression 



Variable 


b 


Beta 


T 


SigT 


Course = STATISTICS 


10.93 


0.81 


4 .78 


w • W w U 


SCHOOL 


-6.93 


-0.73 


-3 .72 


0.000 


SAT MATH SCORE 


-0.02 


-0.68 


-4 • 68 


0 * 000 


Mode « MIXED 


5.00 


0.50 


3*23 


0. 002 


Course = soc iso 


7.23 


0.48 


3 . 09 


0 « 002 


Course = CIS 213 


2.90 


0.24 


1.89 


0.061 


SAT VERBAL SCORE 


0.01 


0.18 


1.82 


0.071 


ACADEMIC STANDING 


0.60 


0.17 


1.62 


0.109 


Mode = ONLINE 


1.50 


0.16 


1.54 


0. 126 


Course « MATH 305 


-0.58 


-0.05 


-0.40 


0.693 


( Constant ) 


26.46 


«M KM Mi 


6.53 


0. 000 



Multiple R = 0.52 Adjusted R Square = 0.21 
DF (10,121) F = 4.53 p = 0.001 

Note: Low Course Rating scores correspond to favorable ratings 
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TABLE 6-6 

Predicting Final Grade for VC Students : Multiple Regression 
^^^^^^^^ b Beta T SigT 



SAT VERBAL SCORE 0.00 0.296 2.21 0.028 

CONVENIENT -0.18 -0.270 -2.07 0.041 



INCREASED MOTIVATION 


-0. 11 


-0.162 


-1, 


.40 


0.165 


ACCESS PROBLEM 


-0. 15 


-0.155 


-1, 


.40 


0. 166 


TOTAL TIMES ONLINE 


0.00 


0.119 


1. 


07 


0.288 


ACADEMIC STANDING 


0.09 


0.099 


0. 


97 


0.337 


ASSIGNMENTS USEFUL 


0.08 


0.098 


0. 


69 


0.490 


MORE INVOLVED 


-0.06 


-0.078 


-0. 


60 


0.552 


EIES EXPECTATIONS 


-0.01 


-0.068 


-0. 


63 


0.531 


ACCESS PROFESSOR 


-0.04 


-0.053 


-0. 


43 


0.669 


SAT MATH SCORE 


0.00 


0.025 


0. 


17 


0.863 


COMMENTS USEFUL 


-0.00 


-0.006 


-0. 


04 


0,967 


(Constant) 


2.61 




2. 


48 


0.015 



Multiple R = 0.49 Adjusted R sq = 0.14 
DP (12,86) F = 2.29 p = 0.001 
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TABLE 6-7 

Predicting Overall vc Rating : Multiple Regression 



Variable 


b 


Beta 


T 


SiqT 


SAT MATH SCORE 


0.01 


0.29 


3.96 


0.053 


CONVENIENT 


-0.92 


-0.28 


-2.65 


0.010 


ACCESS PROFESSOR 


-0.78 


-0.24 


-2.65 


0.010 


MORE INVOLVED 


-0.79 


-0.22 


-2.22 


0.029 


Course = MANAGEMENT 


-2.18 


-0.16 


-0.41 


0.684 


Course = CIS 213 


-2.66 


-0.16 


-0.49 


0.626 


ASSIGNMENTS USEFUL 


-0.42 


-0.11 


-1.08 


0.284 


Course = MATH 305 


-1.67 


-0.10 


-0.31 


0.759 


ACADEMIC STANDING 


-0.46 


-0.10 


-1.06 


0.292 


COMMENTS USEFUL 


-0.35 


-0.09 


-0.97 


0.337 


INCREASED MOTIVATION 


-0.27 


-0. 08 


-0.99 


0.327 


EIES EXPECTATION 


0.05 


0.08 


0.98 


0.332 


TOTAL TIMES ONLINE 


-0.01 


-0.07 


-0.94 


0.351 


Course = soc 150 


-1.45 


-0. 07 


-0.75 


0.455 


Course = STATISTICS 






•DO 




SCHOOL 


-0.46 


-0. 04 


-0.10 


0.921 


SAT VERBAL SCORE 


0.00 


0.02 


0.21 


0.336 


ACCESS PROBLEM 


0.03 


0.01 


0.06 


0.951 


( Constant ) 


24.35 


mm 4M n* 


2.44 


0.017 



Adjusted R sq = 0.67 
8.82 p « 0.001 

2^2 



Multiple R = 0.82 
DF (18,79) F 
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"Course" is a much stronger predictor of differences in course 
outcomes than is mode of delivery. Bound up with course are 
differences in characteristics of the students enrolled, in the 
subject matter and thus content of the experiences, and especially, 
differences in teacher style or skill in various modes. 

Our primary interest in this chapter was in pursuing the 
question of correlates of relatively "good" outcomes in Virtual 
Classroom courses. Some student characteristics, such as Math SAT 
scores, are strong predictors of relatively good outcomes. 
Convenience of access is also very important, as is regular and 
active participation, and a perception of improved access to the 
professor. These latter two variables, while partially related to 
student characteristics such as self-discipline, could also be 
greatly affected by how the instructor conducts the online course. 
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CHAPTER 7 
SUMMARY AND CONCLUSION 

Despite a far-from-perfect implementation, the results of this 
field trial were generally positive, in terms of supporting the 
conclusion that the Virtual Classroom mode of delivery can increase 
access to and the effectiveness of college-level education. 

Let us review the hypotheses and the findings. Originally, 
there was an hypothesis that the mixed mode results would not simply 
represent an "average" of the VC and TC modes, but might have some 
unique advantages and disadvantages. In the following summary, 
results related to this speculation are included in reviewing each of 
the other hypotheses. 

HI: There will be no significant differences in scores measuring 
MASTERY of material taught in the virtual and traditional 
classrooms . 

Finding: No consistent differences, in one of five courses, VC final 
grades were significantly better. 

« 

This hypothesis was tested using a quasi-experimental design which 
compared the midterm exam scores, final exam scores, and final grades 
attained by students in matched sections of five courses, in 
Computer Science, student performance tended to be significantly 
better, on the average, as measured by grades. Though there were no 
statistically significant differences for the two Freshman level 
courses in Sociology and Statistics, these were courses in which many 
students did D or F work in both modes, and the instructors tended to 
feel that the mode further disadvantaged young, poorly motivated 
students with marginal levels of reading, writing, and quantitative 
skills. 
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H2: The hypothesis that writing scores would improve mort, for 
students in a writing course with access to the Virtual 
Classroom than for students in similar courses who did not use 
the system, was NOT supported. 

This may be because the measure used was not reliable or 
detailed enough. It showed no changes for students in a writing 
course in either the face-to-face or partially online modes. 



H3: VC students will perceive it to be superior to the TC on a number 
of dimensions: 



3.1 CONVENIENT ACCESS to educational experiences (supported). 

3.2 Increased PARTICIPATION in a course (supported). 

3.3 Improved ability uo apply the material of the course in new 
contexts and EXPRESS their own independent IDEAS relating to the 
material . 

Findings Increased confidence in expressing ideas was most likely to 
occur in the mixed modes courses. 

3.4 Improved ACCESS to their PROFESSOR (supported), 

3.5 Increased level of INTEREST in the subject matter, which may 
carvy beyond the end of the course. 

Finding: This was course dependent. Though the avsrages for measures 
of increased interest are higher for both the VC and Mixed modes, 
the overall scores are not significantly different. Interest 
Index scores were highest for the VC mode at NJIT and for the 
Mixed mode courses at upsala. 

3.6 Improved ability to SYNTHESIZE or "see connection among diverse 
ideas and information." 

Finding: No significant differences overall mode interacts with 
course . 

3.7 COMPUTER COMFORT- improved attitudes toward the use of computers 
and greater knowledge of the use of computers (supported) , 

3.8 Improved ability to communicate with and cooperate with other 
students in doing classwork (Group COLLABORATION Skills) . 

Findings: Mixed and course-dependent. Though 47% of all students in 
VC and Mixed modes courses felt that they had communicated more 
with other students than in traditional courses, 33% disagreed. 
The extent of collaborative learning was highest in the Mixed-mode 
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courses . 



3.9 Improved Overall QUALITY, whereby the student assesses the 
experience as being "better" than the TC in some way, involving 
learning more on the whole or getting more out of the course 
(supported) . 

Although the "average" results supported most of the above 

predictions, there was a great deal of variation, particularly among 

courses. Generally, whether or not the above outcomes occurred was 

dependent more on variations among courses than on variations among 

modes of delivery. The totally online upper level courses at NJIT, 

the courses offered to remote students, and the mixed mode courses 

were most likely to result in student perceptions of the virtual 

classroom being "better" in any of these senses. 

HA: Those students who experience "group learning" in the virtual 
classroom are most likely to judge the outcomes of online courses 
to be superior to the outcomes of traditional courses. 

Finding: Supported by both correlational analysis of survey data and 
qualitative data from individual interviews. Those students who 
experienced high levels of communication with other students and 
with their professor (who participated in a "group learning" 
approach to their coursework) were most likely to judge the 
outcomes of VC courses to be superior to those of traditionally 
delivered courses. 

H5: High ability students will report more positive outcomes than low 
ability students. 

Finding: Supported for Math SAT scores. Results for Verbal SAT 
scores much more mixed and inconsistent. 

H6: Students with more pc^itive pre-course attitudes towards 

computers in general and towards the specific system to be used 
will be more likely to participate actively online and to perceive 
greater benefits froiu the VC mode (supported) . 

H7: Students with a greater "sphere of control" on both the personal 
and the interpersonal levels will be more likely to regularly and 
actively particpate online and to perceive greater benefits from 
the VC mode. 

Finding: Very weak support in terms of correlations with "Sphere of 
Control" indices from survey data. However, qualitative interview 
data indicate that inability to regularly devote time to online 
activities, to "make themselves" participate regularly when there 
is no externally imposed schedule of class meetings, was a common 
characteristic of students for whom VC outcomes were relatively 
poor. 
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H8: There will be significant differences in process and outcome 
among courses, when mode of delivery is controlled (strongly 
supported. Course is a much stronger source of variance in 
outcomes than is Mode). 

H9J Outcomes for the second offering of a VC course by an instructor 
will be significantly better than those for the first attempt at 
teaching online. 

Findings: Although there was some tendency for this to be true 
results were not consistently better on all measures for all 
second repetitions, other factors, such as lower levels of skill 
or motivation among the students, may come into play. 

Some courses may not be suited to this mode, and a second 

repetition of the totally online mode of delivery would not improve 

matters. The Introductory Sociology instructor came to this 

conclusion, as did the instructor for the required freshman- level 

course in Statistics at Upsala. Both felt that many of the freshmen, 

at least in the "computer-poor" Upsala environment, lacked the skills 

and the self-discipline to benefit from a totally online course. 

However, both instructors felt that the mixed-modes method of 

delivery could be superior, especially for upper-level courses which 

examine a small number of topics in depth. 

"H^^^ ^® significant differences between the Upsala and 
NJIT implementations of the Virtual Classroom, in terms of both 
process and outcomes of the online courses. 

Finding: Supported. Results were better at NJIT for the totally 
online courses. ^ 

A Note on Costs 

It is difficult to say how much it "costs" to offer online 
courses. The problem is with how one accounts for the costs of the 
central computer and its operation and maintenance. For instance, if 
you already have a mainframe and it is already being operated, then 
it really does not "cost" much more to add more users. 

We can say something about the range of costs for the computing 
service. On EIESl, where this experiment was conducted, we were 
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working with a totally dedicated Perkin-Elmer minicomputer. The 
machine cost about $400,000 and its expected life is five years or 
so. There are maintenance costs? the costs of approximately two full 
time technical people to keep the system operating, two full time 
administrative people who provide user support, plus student 
assistants and overhead, what we have done is priced the use of an 
account at a flat fee of $60.00 per month. At this rate, we are 
actually losing some money each year. This is within the context of a 
system with a capacity of 2000 users, in which about half are "free" 
because they are for internal university use. 

EIESl is an outmoded piece of software running on an outmoded 
piece of hardware. The new generation, TEIES, will run on IBM 
mainframes, and will support operating Virtual Classroom 
simultaneously with other applications. The "costs" and "prices" 
depend on the size of machine being used and the pricing strategy 
adopted to cover costs. We need to gain experience with loads and 
capacities on this hardware. What happens is that you get an economy 
of scale that favors the operation of shared utilities. We estimate 
that on an IBM mainframe configuration costing $400,000, the total 
capacity is about 1,000 active accounts. On the other hand, on a 
mainframe configuration costing about $600,000, we estimate that the 
capacity is about 10,000 active accounts, in the former case, 
amortizing the initial costs of the hardware over an expected life of 
ten years, yields a cost of about $w0 a year per student for 
hardware, plus shares of maintenance and operational costs. 
Operational costs depend upon the level of support given to users. 
In the c&&m of the large mainframe, hardware costs amortized over ten 
yearjs would be only about $10 a year per student. 

In fact, the main "casts" of this mode of delivery are the 
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initial efforts by the instructors to prepare and offer a course 
online for the first time. Secondly, it can be costly to provide 
assistants who are available in person or by phone to help at any 
time. Thirdly, for remote students, telecommunications are a high 
part of the cost, with TELENET rates at $9.50 per hour daytimes and 
$3.00 per hour during the evenings, spending lOO hours online for a 
course can add up to a considerable sum. We recommend that students 
bear the costs of telecommunication, just as they bear the costs of 
commuting to a traditional course. This will motivate them to use 
off-peak rather than expensive prime time, and to use uploading and 
downloading to minimize connect time. Another approach is to give 
each student an allocation of "X" free hours; after that, they would 
have to pay for additional hours of use of TELENET or similar 
packet-switched networks to reach the Virtual Classroom. 

one may better understand the elasticity of connect time by 
re-examining the data on connect times by course. The NJIT CIS 
students, who had unlimited connect time, often at 9600 baud on a 
local area network, spent an average of seventy five hours online. 
Each session generally averaged one half hour; obviously, many went 
well over an hour. The Connected Education students, who were 
reaching the Virtual Classroom via TELENET and who had to handle 
their local phone charges to reach a TELENET node, managed to 
complete an entire course with a much lower rate of actual connect 
time: thirteen hours, on the average, with an average session of 
under twenty minutes. 

Thus, one of the strategies for minimizing costs must he to have 
students use a microcomputer for composing and displaying material 
locally, when they are coming into the system remote, rather than 
burning up hours with remote text input. Our new microcomputer 
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package, Personal TEIES, is designed to support a mode of operation 
whereby it is simple and automatic to decide to upload and download 
items between the local PC and the central conferencing system, and 
thus to minimize actual connect time. 



Modes of Use of The Virtual Classroom 
There are several modes of employment of the virtual Classroom. 
It can be used in a "mixed modes" manner on a local campus, to 
support a quarter to three quarters of the coursework for classes 
which also have some face-to-face meetings. This "adjunct" or 
"mixed" mode seems appropriate for a wide range of courses, including 
lower level courses. It can be used to deliver totally online 
courses, to remote or distance education students and/or students 
who are taking other courses at a campus in a traditional classroom. 
For totally online courses, it is recommended that the material be at 
a sophomore or higher level, or else that students be screened very 
carefully, to advise those with poor study skills against an 
introductory course offered online. 

VC can also be used, very fruitfully, for remote education at 
the graduate level, or for continuing professional education of 
employees within organizations. Though not the purview of this 
project, the application area of continuing professional education 
may be the biggest "market" for Virtual Classroom in the long run. 
Such courses -typically enroll mature, motivated students; focus on a 
few related topics; and have students for whom convenience of access 
would be very important. 

The two year program of the Western Behavioral Sciences 
Institute provides one model of the use of the VC for executive 
education. There are four six-month terms, and at the beginning of 
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each term there is a one-week residential seminar in La Jolla. Each 
term is divided into month-long seminars on specific topics, while a 
number of conferences and activities (such as small informal 
discussions groups of about ten) are continuous. At the end of the 
two-year program, about three quarters of the participants elect to 
remain in the network as alumni Fellows. The WBSI president. Richard 
Farson (1987) notes the following major advantages of online 
education: 

A program of depth and intensity, without removing th^ 
executive from his job for extended periods of time.,. 

The network permits the executive to form a genuine 

community on a relatively permanent basis, to sustain 
them th-.oughout their careers. 

Certainly, one aspect of the Connect-Ed and WBSI programs which 
should be emulated in future projects is that students take more than 
a single course online. Just as the instructors tended to improve 
their ability to work in this new environment with repitition, so it 
may be expected that students can improve their ability to use the 
technology effectively on the basis of experience. 

Qualitative Outcomes and Overall Conclusions 
In many cases, results of the quantitative analysis are 
inconclusive in determining which is "better," the VC mode or the TC 
mode. The overall answer is, "it depends." Results are superior for 
well-motivated and well-prepared students who have adequate access to 
the necessary equipment and who take advantage of the opportunities 
provided for increased interaction with their professor and with 
other students, and for active participation in a course, students 
lacking the necessary basic skills and self-discipline will do better 
in a traditionally delivered course. 

The "verdict" on virtual classroom comes down, in the end, to 
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the qualitative reactions of students and instructors who were 



stimulated by this new type of learning environment. For example, 
here is the text of a message from a student in the Management 
Laboratory, sent after the course was officially over; 

X * Roxanne, I just completed Enrico's 471 class here on EXES, 
u f ■ ^^^^ ^ should give you what I feel about the class and 
What It has done, it was the most stimulating, fascinating, 
educational and social experience I have ever had! From the 
subject itself to how it was presented to the activity and 
enthusiasm of this class, it was beyond words. I feel that the 
method of how it was presented here, on the system, had more 
than a great deal to do with it. it also had to do with 
Enrico's abilities as well as a bunch of very energetic people 
who were able to excel in his or her own way thru the extended 
class on the system. 

A lot of what happened, the massive activity in the 
conferences, the massive amount of time spent online by each 
participant, and the new, good and lasting friendships that 
developed ( AND THERE ARE A WT OF THOSE ) will never be given 
justice in whatever the results of this project are, but they 
are what was really meaningful in this course. A great deal of 
learning was accomplished concerning the topic and a lot of 
other ideas. Learning that would not have been so great and 
varied as it was (without the system) . 

I am not the only person who feels this way; its shared by 
most of the class... 



I have never dreaded so much the end of a semester and I 
hope that the group that formed and its cohesiveness that was so 
strong will continue afterwards, i don't want to belabor the 
point, but do want to emphasize what a great thing it was and 
hope to see it continue for a long time to come because the 
quality of the educational experience is greatly increased not 
only for the subject matter, but on a social level as well. 

Thanks for giving us this chance. 

Essentially, that's what the Virtual Classroom software 

provides— a chance to participate in a different kind of learning 

experience, one based on an active learning community working 

together to explore the subject area of a course. Note that the 

Management Laboratory was referred to above as "officially" over. 

Several months after the grades had been turned in, the class 

conference was still active, with over a hundred new entries which 

continued to discuss the issues raised in the course. This type of 
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behavioral indicator of development of a high level of interest in 
learning validates the responses of students to questionnaire items. 

The VC is not without its disadvantages, and it is not the 
preferred mode for all students (let alone all faculty), students 
(and faculty) report that they have to spend more time on a course 
taught in this mode than they do on traditional courses. Students 
also find it more demanding in general, since they are asked to play 
an active part in the work of the class on a daily basis, rather than 
just passively taking notes once or twice a week. For students who 
want to do as little work as possible for a course, the Virtual 
Classroom tends to be perceived as an imposition rather than an 
opportunity. The VC is also not recommended for students who are 
deficient in basic reading, writing, and computational skills. 

We have noted that increased interaction with the professor and 
with other students is the key to superior results in the Virtual 
Classroom. Thus, the selection and orientation of instructors who 
can orchestrate such collaborative learning environments becomes the 
key to success. The second volume of this report focusses on the 
issue of effective online teaching techniques. 
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IDENTIFYING INFORMATION 

pSt^ffiifyini codef ^or\h'r^ questionnaire as soon as we have 
confiSt;Tt?^^ codes on the other pages, in order to protect the 
contidentiaLity of your responses. ^ ^^^^^^ uue 



NAME 



ADDRESS 



CITy, STATE, SIPCGDE„ 
STUDENT ID NUMBER 
HOME TELEPHONE: 
DATE ; 



Al 
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BASELINE QUjSSTIONNAIRE FOR STUDENTS 
VIRTUAL CLASSROOM PROJECT 



COURSE NAME: . 

COURSE NUMBER AND SECTIONT 
lijISTRUCTOR: 



??? T«^°*^® ^^^'^^ was presented X=1.91 SD=0.84 N«37? 

(1) 10* Completely Online ^^^^ 

(2) 28% Pairtially Online 

(3) 32% All Qttlii'm 

I^am?^ X«1.60 SD«0.86 N«332 

(1) 58% An NJIT stud^ant 

(2) 32% upsala st\ident 

(3) _4% New ScJiiciol (Connect Ed) student 

(4) jn Other 

SOME BACKGROUND INFORMATION 

If you feel that any of these items invade your privacy, vou are of 
course free to decline to answer them. Privacy, you are of 



Sr^^U^ • ?^ ^'^^ following reasons for your taking this 

vn^^^?r„*"i,^^^P^'^^°l^^?^ "'Ode of delivery of the course? 

very Important, Somewhat Important, or Not Important? 

Very Somewhat Not 

Important Important Important X SD N 

PROFESSIONAL INTEREST 

LH^''^J.rS^'*^f^°"4 ?^ 32% 46% 22% 1.89 0.73 331 

30b- related interest in 

the topic 
GENERAL INTEREST 

? ? general interest 32%. 57% 10% 1.73 o.'62 329 

in the topic " — ' 

REQUIRED MAJOR 

Required for my major 47% 74% 100% 1.78 0.83 326 

REQUIRED COURSE 

Required for graduation 56% 22% 22% 1.66 0.82 325 

INSTRUCTOR'S REPUTATION 

The reputation of the 2,2% 40% 37% 2. 15 0.76 316 
instructor 

NO CHOICE 

No choice- transfer to 5% 14% 82% 2.77 0.52 303 

Other sections impossible 
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CURIOUS 

I was curious about how 
the technology works 

CONVENIENCE 

More convenient than 

traditional classes 



Very Somewhat Not 
Important Important Important 



32% 



26% 



48% 



33% 



21% 



X SD N 
1.89 0.72 326 



41% 2.15 0.81 318 



EXPECTED GRADE 

What grade do you expect to receive in this Course^ 
55%A39%B6%C^D X=1.51 SD=:.62 



N=321 



EXPECTED DIFFICULTY X=3.44 SD=.90 N=331 

How easy or difficult do you expect this course to be? 



3 : 
39% 

29% Female 



4 
38% 



5 
11% 



EASY : 1 : 2 
3% 10% 

SEX 

Your sex: 71% Male 
AGE 

Your age at last birthday: 
17,18 13% 
19- 10% 
22-25- 27% 
26-34 18% 
35+- 6% 

MAJOR 

Your major: 

NATIONALITY 
Nationality: 

(1) 57% USA 

(2) 43% OTHER 

ETHNIC GROUP 

Ethnic/Racial Background 
14% Black/Afro-American 

_7% Hispanic (Mexice.n, Puerto-Rican, etc.) 
66% White 

ii^ Asian or Asian-American 
_.U Other 

ENGLISH 

Is English your native or first lanquaqe? 
sn Yes 19% No 

TYPING 

How would you describe your typing skills? 



: DIFFICULT 

X=1.29 SD=0.46 N=327 
X=23.77 SD=6.78 N=320 



X=1.43 SD=.50 N=250 



X=1.19 SD=.39 N=325 



X=3.03 SD=.92 N=331 



(^) None 

(2) 22% Hunt and peck 

(3) 46% Casual (rough draft with errors) 
22% Good (can do 25 w.p.m. error free) 

(3) _6% Excellent (can do 40 w.p.m. error free) 
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ACADEMIC STANDING 
Academic standing 



X=2,99 SD=1.31 N=321 



16% Freshman 
20% Sophomore 
31% Junior 
21% Senior 

11% Master's candidate 
2% Doctoral candidate 
1% Post-doctoral 



PREVIOUS ONLINE X=1.15 SD- 47 N-130 

How many online ("virtual classroom") courses have* you taken 
previously? ^ 

(1) 90% None. This is my first online course 

(2) _5% One 

(3) 5% Two or more 

IMAGES OF YOURSELF 

rtf^^^^L''??'* n^""^ °£ ^^''^ following and indicate how much you agree or 
disagree (1= Completexy DISAGREE: 7 means Completely AGREE) . 

DISAGREE AGREE 
1234567XSDN 

WORK HARD 

U'S isK'becaur I °* 8% 21% 36% 30% 5.76 1.15 331 

worked hard for it 

GROUP EASY 

L^ill?-^^ ^° P^^^ ^" ^* 24% 28% 20% 11% 4.75 1.38 329 

important part in most 

group situations 
PREFER LUCK 

I prefer games involving 14% 19% 18% 22% 14% 8% 4% 3.43 1.66 326 
some luck over games 
requiring pure skill 

POOR SOCIAL CONTROL 

s:u4Sn?id:nt':?i^2'„ost " 
things, I still seem to 
lack the ability to 
control social situations 

LEARN ANYTHING 

inS?Mi«^f!? . . °* 4* 30% 48% 6. 17 1.04 330 

anything if I set my mind 

to it 

MAKING FRIENDS 

LU'^l^L!!? ^'•^^^le "laking 0% 1% 4% 8% 17% 27% 43% 5.93 1.22 328 

ana keeping friends A4 
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DISAGREE AGREE 
^■234567XSDN 

POINTLESS 

inr2<S«^!!^^!«L^?.^®®^u . ^"^^ 29% 13% 13^\ 8% 5% 4% 2.80 1.70 328 
working on something that 

is too difficult for me 
CONVERSATIONS 

I'm not good at guiding 22% 25% 17% 15% 12% 6% 2% 2.95 1.61 329 
the course of a 
conversations with several 
others 

COMPARISONS 

oLS^^S??^ ?^^^^^''^ ®* 27% 24% 4.99 1.86 328 

competition I like to know 

how well I do relative to 

everyone else 

CLOSE RELATIONSHIPS 

I can usually establish a 5% 2% 9% 18% 21% 24% 21% 5.07 1.60 327 
close personal 
relationship with someone 
I find attractive 

ABILITY 

My major accomplishments 0% i% 2% 6% 20% 37% 34% 5.92 i.06 328 
are entirely due to my 
hard work and ability 

MAKING PLANS 

When I make plans I am 0% 2% 4% 14% 28% 31% 21% 5.43 1.22 330 

almost certain to make 
them work 

STEER INTERVIEWS 

When being interviewed I 

can usually steer the 3% 7% 15% 29% 23% 15% 6% 4.33 1.43 326 

interviewer toward the 
topics I want to talk 
about and away from those 
I wish to avoid 

SETTING GOALS 

I usually don't set goals 32% 34% 16% 8% 5% 3% 1% 2.34 1,41 328 
because I have a hard time 
following through on them 

GETTING HELP 

HJ carrying 21% 24% 21% 17% 8% 7% 2% 2.94 1.57 327 

off a plan of mine, it's 
usually difficult to get 
others to help 

COMPETITION 

Competition discourages 47% 20% io% 9% 7% 3% 3% 2,32 1.68 329 
excellence 
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DISAGRi^E AGREE 
1234567XSDN 

MEETING PEOPLE 

" »eeri%ansSIny"'"'^ 
arrange it 

OTHERS LUCKY 

Other people get ahead 22% 26% 17% 20% s% 3% 3% 2.88 1.55 328 

just by being lucky ^,00 x.oo j-io 

POINT OF VIEW 

I often find it hard to 20% 29% 20% 15% 9% 4% 2% 2.84 1.53 330 
get my point of view ^-5 ^.o'* x.dj jjo 

across to others 



DISAGREEMENTS 
In attempting 
over a disagr 
usually make it worse 



In attempting to smooth 30% 31% 18% 13% 5% 1% 2% 2.45 1.42 327 
over a disagreement I -» x.h^ jz/ 
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COMPUTER EXPERIE^?f HXPE.IENCE WITH COMPUTERS 

cSipStIr sSlteSsf"'"" "^'^ '^^""^'"^^ experience "ith' 

(J) 12% I am a NOVICE; seldom or never use computers 
? Ml J OCCASIONALLY used computer terminals and systems before 

(3) 22% I have FREQUENTLY used computer systems y*»^eins oerore 

(4) 11% Use of computers is central to my PROFESSIONAL work 

following pairs of words, please circle the 
^nS?SS?po ^ IS closest to your CURRENT FEELINGS ABOUT USING 
?2Too™nn^/°'' instance, for the first pair of words, if you 
f nd no? ??^?! ''^hS^?? ^?"®f^^ completely "stimulating" to use 

2^^^ "dull," circle "i"; "4" means that you are 
undecided or neutral or think they are equally likely to be 
stimulating or dull; "3" means you feel that they are slightly more 
stimulating than dull, etc. s»xayntj.y more 



DULL-l 
Stimulating 



DREARY- 1 
Fun 



DIFFICULT- 1 
Easy 



IMPERSONAL- 1 
Personal 



HELPFUL- 1 
Hindering 



1, 2 3 4 5 6 7 Dull 
23% 24% 21% 21% 5% 2% 3% 



12 3 4 5 6 7 Dreary 
22% 27% 23% 15% 8% 2% 3% 



1^ 2 3 4 5 6 7 Difficult 
7% 15% 18% 27% 16% 12% 5% 



12 3 4 5 6 7 Impersonal 
6% 10% 13% 36% 11% 13% 11% 



1„ 2 3 4 5 6 7 Helpful 
4% 2% 5% 15% 16% 31% 27% 



UNTHRE»r2NING-l 
Threatening 1 



3 4 5 6 7 Unthreatening 



4% 6% 6% 26% 12% 21% 26% 



INEFFICIENT-1 
Efficient i 



7 Inefficient 



OBLIGING-1 
Demanding 



38% 30% 15% 10% 2% 2% 2% 



12 3 4 5 6 7 Obliging 
12% 12% 13% 40% 11% 8% 4% 



UNRELIABLE-1 

Reliable 12 3 4 5 6 7 Unreliable 

24% 27% 22% 18% 4% 2% 3% 

UNDESIRABLE- 1 

Desirable 12 3 4 5 6 7 Undesirable 

25% 26% 16% 23% 3% 3% 4% 
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X SD N 

2.82 1.52 325 

2.78 1.49 327 

3.82 1.57 327 

4.20 1.63 324 
5.35 1.58 323 
5.02 1.68 325 

2.21 1.37 323 
3.65 1,54 323 
2.70 1.46 326 
2.77 1.57 327 



EXPECTATIONS ABOUT THE EIES SYSTEM 
[Skip this section if you are not going to use EIES] 

Indicate your expectations about how it will be 1o use this system 
by circling the number which best indicates where your feelincs lie 
on the scales below. ^ 



EASY-l 

4% 6% 
: 1 : 2 
Hard to 
learn 



FRIENDLY- 1 

4% 7% 
: 1 : 2 
Impersonal 



14% 
3 



25% 
4 



19% 
5 



20% 
6 



X«4.54 SD=1.58 N=:246 



8% 
3 



24% 
4 



28% 
5 



20% 

6 



X=4.60 SD=1.52 N=:244 



NOT FRUSTRATING- 1 

4% 10% 16% 

: 1 : 2 : 3 
Frustrating 



24% 
4 



21% 
5 



21% 
6 



X=4.32 SD=1.59 N=245 



PRODUCTIVE-1 

2% 1% 
: 1 : 2 
Unproductive 



5% 

3 



18% 
4 



24% 
5 



34% 
6 



X~5.27 SD=1.29 N=244 



11% 
7 : 
Easy to 
learn 



9% 

7 : 
Friendly 



9% 

7 : 

Not 
frustrating 



16% 
7 : 

Productive 



EFFICIENCY- 1 

Do you expect that use of the System will increase the efficiency of 
time)? " quantity of work that you can complete in a given 



19% 21% 
! 1 : 2 
Definitely 
yes 



14% 
3 



24% 
4 



15% 
5 

Unsure 



5% 
6 



X=3.00 SD«1.55 N=245 



2% 

7 : 

Definitely 
not 



QUALITY- 1 

Do you expect that use of the System will increase the quality of 
your education? 



21% 22% 
! 1 : 2 
Definitely 
yes 



18% 25% 
3 : 4 
Unsure 



6% 
5 



4% 
6 



X=5.48 SD=1.74 N=242 



3% 

7 : 

Definitely 
not 
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RESENT-1 

I resent being required to use EIES for this course. 



4% 3% 6% 19% 7% 17% 43% 

:l; 2:3:4:5:6:7: 
Definitely Unsure Definitely 

yes not 
X=2.76 SD=1.46 N=243 

OVERALL- 1 

Overall, how useful do you expect the System to be for online classes? 

23% 27% 20% 19% 6% 3% 2% 

:l:2:3:4:5:6:7: 

Very Not useful 

Useful at all 

X=3.37 SD=1.08 N=237 

EXPEC'^.^ED TIME X==3.37 SD=1.08 N=237 

While you are part of an online course, 

how much time in the average week do you foresee yourself using EIES 
in relation to your coursework? 

(1) 4% Less than 30 minutes 

(2) 12% 30 minutes to l hour 

(3) 43% 1-3 hours 

(4) 29% 4-6 hours 

(5) J7% 7-9 hours 

(6) 5% 10 hours or more 

EQUIPMENT ACCESS 

Please describe your access to a computer terminal or microcomputer 
at your office or place of work. 



WORK ACCESS X=3.00 SD=1.66 N=264 

(1) m No terminal 

(2) 21% Have my own terminal 

(3) 10% Share a terminal, located where I can see it from my desk 

(4) _8% Share a terminal, which takes minutes to reach 

(5) 32% Not applicable; I do not have an office 

HOME ACCESS X=1.41 SD=0.49 N-267 

Do you have a micro or terminal at home (or in your dorm, wherever 
you live during classes)? 

(1) 59% No 

(2) 41% Yes 

TERMINAL TYPE X=2.04 SD=0.94 N=200 

What kind of terminal do you usually use? (Check all that apply) 

42% CRT (video display) 

11% Hard copy (printer terminal) 

46% Both 
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MICRO 

4p% Microcomputer (Brand: ) 

25% With modem 
26% With hard copy 
34% With disk storage 

If you know the name of your communications softwar^j fe.a. 
Smartcom) , please list it here:_ ' 

THANK YOU VERY MUCH » ! I 
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POST-COURSE QUESTIONNAIRE FOR STUDENTS 
VIRTUAL CLASSROOM PROJECT 

COURSE NAME! 



COURSE NUMBER AND SECTION: 

INSTRUCTOR : 

YOUR STUDENT ID: ~ 



COURSE EFFECTIVENESS 

^r. J^^l^ ^!?^ee sets of items in this section; we would like vou 

the LTnH^T''^^^ thinking. The firsr relates ^to 

Inltr-ultnf^^L presentation style and effectiveness of your 
oStoomo«°''L^?L ^e°°"<*', to the course content; and the third, to ?he 
Sho SI?ticwL°?n''r ^?^- ^^^^^ questionnaire thosi 

^SSpa?L^^i^tt^ direct 

corres^o^dHo tSe%^?Lwfn1^°:^?I; ''^^'^ ^^^^^ ^ ^^^^ 

SAa Strongly Agree 
A« Agree 

N= Neither agree nor disagree (neutral) 
D« Disagree 
SD= Strongly Disagree 

COURSE CONTENT 



SA 



CONTENT INTERESTING 
The course content was 
interesting to me 

CONTENT IMPORTANT 

Course content is important 

or valuable 

GOALS CLEAR 

Course goals were clear to 
me 



N 



D SD 



X 



SD 



N 



20% 63% 12% 4% 0% 2.01 0.72 283 



25% 58% 14% 2% 1% 1.96 0.74 283 



16% 59% 19% 6% 1% 2.18 0.80 282 



REQUIREMENTS CLEAR 
Work requirements and 
grading system were clear 
from the beginning 



26% 46% 19% 6% 2% 2.11 0.93 283 



READINGS POOR 

The reading assignments are 4% 
poor 

WRITTEN ASSIGN. POOR 

The written assignments are 2% 

poor 

LECTURES POOR 

The lecture material is poor 2% 

All 



8% 25% 48% 15% 3.63 0.96 283 



4% 28% 49% 17% 3.74 0.87 281 



5% 14% 51% 27% 3.95 0.92 279 



2ra 



SA A N D SD X SD N 

WORK HARD 

The^students had to work 18% 45% 29% 7% 1% 2.28 0.88 283 

WASTE OF TIMu; 

time ^ 2* 14% 32% 49% 4.21 0.96 282 

APPROPRIATE LEVEL X=3 ^ 18 SD==0.63 N=280 

Is this course taught at an appropriate level? 

1% 8% 68% 21% 3% 

: 1 : 2 : 3 : 4 . 5 . 
Too easy just right Too difficult 

COURSE OVERALL X=2,48 SD=0.97 N=265 

How would you rate this course over-all? 

(l)Excellent (2)Very good (3)Good (4)Fair |5)Poor 
16* 37% 34% 11% 3% 

COMMENTS ABOUT THE COURSE CONTENT? 
Yes Comment : 16% 
No Comment : 84% 



CHARACTERISTICS OF THE TEACHING 

SA A N D SD X SD N 

WELL ORGANIZED 

Instructor organized the 31% 55% io% 2% 1% 1.89 0.79 280 

course well 

GRADING FAIR 

Grading was fair and 29% 50% 18% 2% 1% 1.97 0.80 276 

impartial 

ENJOYS TEACHING 

Instructor seems to enjoy 50% 39% 9% i% o% 1.64 0.74 277 
teaching 

LACKS KNOWLEDGE 

Instructor lacks sufficient 2% 4% 5% 29% 59% 4.38 0.95 279 

knowledge about the subject 

area 

IDEAS ENCOURAGED 

Students were encouragei to 40% 48% 9% 3% 0% 1.74 0.73 280 
express ideas 

PRESENTED CLEARLY 

Instructor presented 27% 55% 14% 3% 1% 1.95 0.79 280 

material clearly and 
summarized main points 
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SA A N D SD X SD N 

OTHER VIEWS 

Instructor discussed points 25% 52% 20% 4% o% 2.02 0.77 279 

of view other than her/his 

own 

PERSONAL HELP 

The student was able to get 27% 45% 23% 3% 1% 2. 06 0.86 278 
personal help in this course 

INSTRUCTOR BORING 

Instructor presented 2% 6% 21% 45% 26% 3.85 0.95 277 

material in a boring manner 

HELPFUL CRITIQUE 

Instructor critiqued my 17% 48% 30% .1% 2% 2.25 0.84 279 

work in a constructive and 
helpful way 



TEACHER OVERALL 

Overall, I would rate this teacher as 



X=1.87 SD=0.90 N=279 



(l)Excellent (2)Very good (3)Good (4) Fair (5) Poor 
40% 38% 16% 4% 1% 

COMMENTS ABOUT THE INSTRUCTOR OR THE TEACHING? 
Yes Comment : 26% 
No Comment : 74% 
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OUTCOMES OF THE COURSE 

SA A N D SD X SD N 

MORE INTERESTED 

I became more interested in 18% 52% 21% 6% 2% 2.22 o.90 283 
tne sub J ect 

LEARNED FACTS 

?actSartta?e?iar "^^^ °' "* 5% i% 2.20 0.74 283 

CONCEPTS 

I i^ained a good 16% 68% 11% 4% 1% 2.05 0 71 pr? 

understanding of basic 

concepts 

CENTRAL ISSUES 

I learned to identify 12% 61% 22% 3% 2% 2. 21 0.76 281 

central issues in this field 

COMMUNICATED CLEARLY 

I developed the ability to 13% 50% 31% 3% 2% 2.30 0.81 283 
communicate clearly about 
this subject 

CRITICAL THINKING 

My skill in critical 12% 50% 32% 5% 2% 2.34 0.82 283 

thinking was increased 

ETHICAL ISSUES 

Lrsfflng"of ethical 
issues 

GENERALIZATIONS 

My ability to int grate 10% 51% 30% 7% 1% 2.29 0.82 280 

facts and develop 
generalizations improved 

COMPLETED READINGS 

I regularly completed the 20% 43% 23% 12% 3% 2.35 1.02 280 
required readings 

DID ADDITIONAL READING 

^° 23% 42% 23% 5% 2.98 0.97 282 

additional reading 
PARTICIPATED 

I participated actively in 18% 42% 30% 8% 1% 2.32 0.91 279 
class discussion 

DISCUSS OUTSIDE 

I was stimulated to discuss 12% 38% 32% 16% 2% 2.58 0.96 283 

related topics outside of 

class 
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WRITTEN AIDED 

a$der»"!,aS""'"'' "* 

COMPLETED WRITTEi>[ 

irSK^LSS'' '-'^ ^" 

THINK FOR SELF 

myse?/"''*'^^ ^^^""^ 1-93 0.69 283 

EXPRESSING IDEAS 

I became more confident in 18% 47% 30% 3% 1% 2.23 o 83 5fli 
expressing my ideas ^'^^ 

NEW FRIENDSHIPS 

in'^this^'clasr'' 51% 21% 5% 4% 2.25 0.96 283 

VALUE OTHERS VIEWS 

pointrS vLr'"^ ^* 2* 2-27 282 

DID BEST WORK 

bes^wS^k^''^^^^ ^° ^* 1* 2.12 0.84 283 

SELF UNDERSTANDING 

I gained a better 10% 39% 43% 5% 4% 2 53 o a? of^^ 

understanding of myself °*®^ 

COMPUTER COMPETENCE 

wit^o^pu^ers =°"P^*^^"<=« 18* 24% 8% 8% 2.45 l.U 281 

RELATIONSHIPS 

«iSships":t„een 
important topics and ideas 



CRITICAL. ANALYSIS 
My ability t 
analyze writ 
was improved 



My ability to critically 10% 45% 36% 7% 1% 2.46 0 82 283 

analyze written material 
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GENERAL INFORMATION 



TOTAL TIME 

About how much TOTAL timo have you spent each week on this course? 
(including "in class" and out, reading and writing, on and offline) 

(1) -II Less than one hour 

(2) 11% 1-2 hours 

m To ^= 275 Mean= 3.6 30= 0.9 

(*•) 38% 5-9 hours 

(5) 16% Ten hours or more 
EASY COURSE 

How easy or difficult was this course for you? 

3% 15% 46% 28% 7% 

EASY: 1:2:3:4:5 : DIFFICULT 

N« 274 Mean= 3.2 SD= 0.9 

EXPECTED GRADE 

What grade do you expect to receive in this course? 
36% A 43% B 16% C _4% D _g% F 

N= 273 Mean= 1.9 SD= 0.8 



Individual vs. Group Learning 

Some courses are essentially a very INDIVIDUAL experience; contact 
with other students does not play an important part in your learning, 
in other courses, communication with other stuaents plays a don^nant 
role. For THIS COURSE, please circle the number below that seems to 
be what you experienced. 

GROUP EXPERIENCE 



10% 
: 1 : 
Individual 
experience 



16% 
2 



21% 
3 



16% 
4 



23% 
5 



N« 266 Mean= 3.6 SD= 1.6 



12% 
6 

Group 

experience 



MISLEADING HELP 

The help I got from other students was— - 



6% 26% 
; 1 : 2 : 
Crucially important 
to me 

N= 274 



36% 
3 



17% 
4 



11% 
5 



Mean= 3.1 SD= 1.2 



5% 
6 

Useless or 
misleading 
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students in my class tended to be 



STUDENTS COOPERATIVE 



29% 
4 



1% 6% 16% 

' 1 : 2 : 3 : 
Not at all 
cooperative 

N« 273 Mean= 4.3 SD= l.i 
STUDENTS COMPETITIVE 

4% 16% 23% 

J 1 : 2 : 3 : 
Not at all 

competitive 

N= 257 Mean= 3.6 SD= 1.2 



34% 
5 



34% 
4 



18% 
5 



15% 
6 

Extremely 
cooperative 



5% 
6 

Extremely 
competitive 



STUDENT COMMUNICATION 

How often did you communicate with other students outside of class 
by computer, "face-to-face" or on the telephone? ^''^''''^ °^ 



11% 
: 1 
Never 



20% 
2 



19% 
3 



27% 
4 



18% 

5 



N= 274 Mean= 3.4 SD== 1.4 



6% 
6 

Constantly 



ATTITUDES TOWARD COMPUTERS" 

For each of the following pairs of words, please c'rcle i-he 
response that represents where you fall on the sc4le in terSs of von^ 
CURRENT FEELINGS ABOUT USING COMPUTERS . 



DULL 

22% 26% 
: 1 : 2 
Stimulating 



24% 
3 



16% 
4 



6% 
5 



N= 265 Mean= 2.8 SD= 1.5 



4% 
6 



3% 
7 

Dull 



DREARY 



22% 

1 
Fun 



22% 
2 



28% 
3 



14% 
4 



7% 
5 



N= 265 Mean- 2.9 SD= 1.5 



4% 
6 



3% 

7 : 
Dreary 



DIFFICULT 



11% 
1 

Easy 



13% 



19% 



22% 
4 



20% 
5 



N- 2 66 Mean= 3.8 SD- 1.7 



9% 

6 



6% 



Difficult 
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IMPERSONAL 



9% 11% 
1:2 
Personal 



HELPFUL 

3% 4% 
: 1 : 2 
Hindering 



UNTHREATENING 

3% 6% 
: 1 : 2 
Threatening 



19% 
3 



28% 
4 



15% 
5 



262 Mean= 3.9 SD= 1.7 



9% 
3 



14% 
4 



18% 
5 



N= 265 Mean= 5.2 SD= 1.6 



10% 
3 



20% 
4 



16% 
5 



N= 264 Mean= 5.0 SD= 1.7 



INEFFICIENT 

28% 
: 1 : 
Efficient 



OBLIGING 



27% 
2 



19% 
3 



17% 
4 



3% 

5 



N= 263 Mean= 2.6 SD= 1.4 



9% 
6 



29% 
6 



21% 
6 



2% 
6 



10% 
7 : 
Impersonal 



23% 
: 7 
Helpful 



24% 
; 7 : 
Unthreatening 



3% 

7 : 
Inefficient 



12% 14% 20% 

: 1 : 2 : 3 
Demanding 



30% 
4 



UNRELIABLE 

18% 30% 18% 

: 1 : 2 : 3 
Reliable 



20% 
4 



10% 
5 



N= 261 Mean= 3«6 SD= 1.6 



UNDESIRABLE 

27% 22% 
: 1 : 2 
Desirable 



16% 
3 



20% 
4 



6% 
5 



N= 262 Mean= 2.9 SD= 1.5 



7% 
5 



N= 264 Mean-- 2.9 SD= 1.7 



10% 
6 



6% 
6 



4% 

6 



5% 



Obliging 



2% 

: 7 : 
Unreliable 



4% 

7 : 
Undesirable 
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ATTITUDES TOWARD MEDIA 
To what extent do you agr©e with the following statements? 
ENJOY LECTURES 

I enjoy listening to lectures. 



7% 25% 
: 1 : 2 
Strongly 
Agree 



LIKE READING 
I like to read. 



26% 
3 



21% 
4 



13% 
5 



6% 
6 



N- 87 Mean« 3.3 SD= 1,4 



Strongly 
Disagree 



10% 
: 1 
Strongly 
Agree 



20% 
2 



25% 
3 



25% 
4 



5% 

5 



9% 
6 



N~ 87 Mean= 3.4 80= 1.6 



6% 
: 7 
Strongly 
Disagree 



DIFFICULTY WRITING 

I have difficulty expressing my ideas in writing. 

2% 9% 15% 

: 1 : 2 : 3 
Strongly 

Agree 



13% 
4 



20% 
5 



28% 
6 



N= 86 Mean= 4.7 SD= 1.6 
LIKE DISCUSSION 

I like to take part in class discussions. 



13% 
: 7 I 
Strongly 
Disagree 



17% 30% 
: 1 : 2 
Strongly 
Agree 



16% 
3 



17% 
4 



9% 
5 



8% 

6 



N= 86 Mean- 3.0 SD= 1.6 



1% 

: 7 : 
Strongly 
Disagree 
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PARTICIPATION IN THE ONLINE COURSE 

If you participated in a traditional course or a course which d^d not 
include any online work, skip the rest of the questionnaire. 

ACCESS PROBLEM 

you?°°^^^ to a terminal or micro for the online class a problem for 

7% 15% 19% 20% 39% 

i;l:2: 3:4:5: 
Seriouj Not a 

Problem Problem 
N« 176 Mean= 3.7 SD= 1.3 

BUSY LINES 

How much problem have you had with "busy" lines or no available oorts 
to EXES? ^ 

6% 13% 23% 20% 38% 

• X : 2 : 3 : 4 : 5 : 
Serious Not a 

Problem Problem 
N= 173 Mean= 3.7 SD= 1.3 

SLOW RESPONSE 

To what extent has the slow response of the EIES system been a 
problem or barrier for you? 



14% 
: 1 
Serious 
Problem 



19% 

2 



28% 
3 



22% 
4 



17% 
; 5 
Not a 
Problem 



N= 174 Mean= 3.1 SD- 1.3 

EXPERIENCES WITH EIES 
Indicate your experiences with using this system 

by circling the number which best indicates where your feelinqs lie 
on the scales below. 



EASY TO LEARN-2 



2% 
: 1 
Hard to 
learn 



FRIENDLY-2 



6% 
2 



12% 
3 



9% 
4 



15% 
5 



35% 
6 



N= 176 Mean= 5.2 SD= 1.6 



20% 
7 : 
Easy to 
learn 



5% 

: 1 : 
Impersonal 



2 



10% 
3 



12% 
4 



19% 
5 



31% 
6 



N- 176 Mean- 4.8 SD= 1,7 



15% 



Friendly 
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NOT FRUSTRATING-2 



4% 14% 

: 1 s 2 
Frustrating 



PRODUCTIVE-2 

3% 3% 
: 1 : 2 
Unproductive 



13% 
3 



18% 
4 



17% 
5 



. 23% 
6 



N« 176 Mean= 4.4 



SD= 1.7 



10% 
7 : 
Not 
frustrating 



8% 
3 



16% 
4 



20% 
5 



32% 
6 



N- 176 Mean= 5.1 SD=1.5 



16% 
: 7 : 
Productive 



INCREASE EFFICIENCY-2 

Did use of the System increase the efficiency of your education (^h^ 
quantity of work that you can complete in a given ?ime^? ^ 



11% 18% 
! 1 : 2 
Definitely 
yes 



15% 23% 10% 

: 3 : 4 : 5 : 

Unsure 

N= 175 Mean= 3.7 SD= 1.8 



15% 6% 
6:7: 

Definitely 
not 



INCREASE QUALITY~2 

Did use of the system increase the quality of your education? 



12% 22% 
1:2 
Definitely 
yes 



22% 
3 



22% 

4 : 
Unsure 



8% 
5 



N= 175 Mean= 3.4 SD= 1.7 



6% 7% 
6:7: 

Definitely 
not 
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COMPARISON TO TRADITIONAL CLASSROOMS 



Please compare online "classes" to your previous experiences with 
face to face" college- level courses. To what extent do you 
agree with the following statements about the comparative process 

vou portion of a course in which 

you participated? (circle a number on the scales.) 

CONVENIENT 

Taking online courses is more convenient. 

25% 24% 16% 12% 8% 8% 7% 

•1:2:3:4j5:6:7: 
Strongly Strongly 

A^^®® Disagree 
N« 175 Mean= 3.7 SD= 1.8 

INHIBITED 

I felt more "inhibited" in taking part in the discussion. 

4% 10% 13% 29% 10% 21% 13% 

!1:2:3:4:5:6:7: 
Strongly Strongly 
*^''3^^^ Disagree 
N= 179 Meaii-- 4.5 SD= 1.7 

LESS WORK 

I didn't have to work as hard for online classes. 

3% 9% 9% 16% 20% 24% 19% 

•^•2:3:4:5:6:7: 
ft^°"9ly Strongly 

Disagree 

N= 181 Mean= 4.9 SD- 1.7 
COMMUNICATED MORE 

I communicated more with other students in the class as a 
result of the comptiterized conference. 

13% 2:.% 13% 19% 10% 12% 11% 

•lJi s3:4:5:6:7: 

Strongly 
Disagree 

N« 179 Mean= 3.7 SD= 1.9 
ACCESS PROFESSOR 

Having the computerized conferencing system available provided 
better access to the professor ( s) . 

18% 21% 19% 14% 10% 9% 8% 

-2:3:4:5:6:7: 
Strongly Strongly 

A^^®® Disagree 
N«179 Mean= 3.3 SD= 1.9 




INCREASED MOTIVATION 

TIlny.-.f«fH ^^^^ n»y assignments would be read by the other students 
increased my motivation to do a thorough job. 

16% 25% 14% 20% 6% 11% 8% 

;1 22:3:4:5:6:7. 
Ccrongly «*. •, ' 

Agree^ ^ Strongly 

N« 179 Mean. 3.4 SD= 1.8 
STOP PARTICIPATING 

When I became very busy with other things, I was more liKely to stoo 
weekly face-to-face lecture. 

15% 20% 14% 15% 8% 15% 14% 

''••2:3:4:5:6:7: 

XT . Disagree 

N= 177 Mean= 3.8 SD= 2.0 

MORE BORING 

clLses!"^ °^ virtual classroom mode is more boring than traditional 

B% 6% 8% 16% 17% 24% 21% 
Lr-nLiJ 2:3:4:5:6:7: 

Agrle''^ Strongly 

^ Disagree 

N= 177 Mean= 4.8 SD- 1.8 :»ayiree 

MORE INVOLVED 

I felt more "involved" in takii.g an active part in the course. 

16% 22% 18% 18% 13% 6% 6% 

Lr^J^r.^./' 2:3:4:5:6:7: 

Agree"'" ^n^-"^^^ 
KT „ Disagree 

N= 177 Mean- 3.3 SD= 1.7 

COMMENTS USEFUL 

I found the comments made by other students to be useful to me. 

11* 28% 20% 20% 10% 7% 4% 

•1:2:3:4:5:6:7: 

Agree"'" ^^-"^1^ 
^ Disagree 
N= 177 Mean= 3.3 SD- 1.6 

ASSIGNMENTS USEFUL 

iLfSU?^..^®^'^^"*^ reviews or assignments of other students to be 
userui to me. 

12% 24% 26% 21% 6% 7% 4% 

•1*2:3:4:5:6:7. 

^'^^^^ „ Disagree 

N= 177 Mean:* 3.2 SD= 1.6 
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NOT CHOOSE ANOTHER 

I would NOT choose to take another online course. 



11* 9% 6% 11% 11% 19% 34% 

, •ls2:3:4:5;6:7. 

Aaree'^'^ Strongl^ 
^ 1.T Disagree 

N« 176 Mean« 4.9 SD= 2.1 

BETTER LEARNING 

I found the course to be a better learning rxperience than normal 
face-to-face courses. 

17% 16% 14% 25% 8% 10% 10% 

•152:3:4:5:6:7: 
Strongly Strongly 

Disagree 

N= 177 Mean= 3.6 SD= 1.9 

LEARNED MORE 

I learned a great deal more because of the use of EIES. 

11% 20% 14% 27% 9% 10% 8% 

Lr'nL^J 2:3:4:5:6:7: 
Strongly Strongly 

Disagree 

N= 176 Mean= 3.7 SD= 1.8 
TRADITIONAL MORE 

I would have gotten more out of a traditional course. 

12% 8% 4% 19% 16% 18% 22% 

•1j2:3: 4:5:6:7* 
Strongly Strongly 

Disagree 

N= 67 Mean= 4.6 SD= 2.0 
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OVERALL COMMENTS AND SUGGESTIONS 
you\ike the^'best?^"^^ virtual classroom experience did 



(1) 34% CONVENIENCE 

(2) 16% ENJOY COMPUTERS 

(3) _9% COMMUNICATE EASY 

(4) 10% CLASS INTEREST 

(5) ^ HARD COPY 

(6) _1% READ HELPFUL N= 119 

(7) 14% SHARE W/OTHERS 

(8) _2i CATCH-UP EASY 

(9) -2% SAY ANYTHING 
(1-0) -2% SELF-PACED 
(11) U ACCOMPLISH MORE 



?{;f "So^c.?^*'^S.^^^T.^^°''^.^°^^^^^^^^^ Classroom experience were 
the "worst," the most in need of improvement? 



(1) 


33% 


SLOW EIES 


(2) 


14% 


NO ACCESS 


(3) 


4% 


HATE COMPUTERS 


(4) 


4% 


NO HELP 


(5) 


3% 


TIME CONSUMING 


(6) 


3% 


NEED DOCUMENTATION 


(7) 


5% 


HATE SELF-PACED 


(8) 


4% 


TOO MUCH WORK 


(9) 


' 3% 


MORE COORDINATION 


(10) 


4% 


TOO HARD 


(11) 


1% 


NO CATCH -UP 


(12) 


1% 


LESS MATERIAL 


(13) 


13% 


BRANCH PROBLEMS 


(14) 


3% 


OTHERS COPIED 


(15) 


2% 


TIME TESTS 


(16) 


3% 


MORE TRAINING 


(17) 


2% 


POOR GRAPHICS 



N= 103 



Other comments or suggestions for improvements? 

(1) ^ REDUCE WORK 

(2) _7% EIES RESPONSE 

(3) _9% MORE ONLINE 

(4) 16% MORE TERMINALS 

(5) _J% HELPS INDEPENDENCE 

(6) -dl IMPROVES PEER RELATIONSHIPS 

(7) 11% HINDERS INDEPENDENCE 

(8) 11% NEED FACE-TO-FACE 

(9) _J% HARD COPY n= 45 

(10) 20% IMPROVE BRANCH 

(11) _2% MORE DOCUMENTATION 
(12; _4% OTHERS SHOULD READ 

(13) _2% IMPROVE SCREENS 

(14) _2% STANDARDIZE SOFTWARE 
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VIRTUAL CIJVSSROOM SOFTWARE FEATURES 



How valuable or useless - and how well designed - do you currpntlv 
find each of the following features or capabilitias of EIES for 
online classes? (If you have not actually used a feature, please 
check "cannot say" and skip to the next feature.) use the space by 
each feature for any cvomments or suggestions. 



PEN NAMES 



Comments 



10% 
1 1 I 
Valuable 



16% 

; 1 : 

Well 
Designed 



25% 
2 



31% 
2 



21% 
3 



40% 

3 



N= 122 Mean= 2.6 



6% 
4 



7% 
5 



N= 165 Mean= 2.7 SD=1.2 



31% 



Useless Cannot 
Say 



8% 5% 
4:5: 
Poorly 
Designed 
SD=l.o 



BRANCH- RESPONSE 

21% 



15% 
1 ; 
Valuable 



12% 
1 

Well 
Designed 



2 



18% 
2 



20% 
3 



15% 
4 



8% 

5 



N= 164 Mean= 2.7 SD= 1.2 



32% 
3 



18% 
4 



21% 



Useless Cannot 
Say 



N= 131 Mean= 3.2 SD= 1.3 



20% 
5 : 
Poorly 
Designed 
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BRANCH- READ 



10% 
: 1 
Valuable 



12% 

1 ; 
Well 
Designed 



21% 
2 



23% 
2 



17% 
3 



10% 
4 



4% 

5 



N- 163 wean* 2.5 SD=^ l.i 



37% 
3 



39% 



Useless Cannot 
Say 



N« 101 Mean= 2.9 



19% 10% 
4:5: 
Poorly 
Designed 
SD= 1.1 



QUIZ 



38% 
: 1 
Valuable 



44% 

1 

Well 
Designed 



19% 
2 



27% 
2 



6% 

3 



20% 
3 



2% 
4 



0% 

5 



N» 64 Mean= 1.6 SD= 0.8 



7% 
4 



36% 



Useless Cannot 
Say 



N= 41 Mean= 2.0 SD= 1.1 



2% 

5 : 
Poorly 
Designed 



RUNNING FORTRAN OR PASCAL COMPILERS 



6% 

: 1 : 
Valuable 



21% 
1 

Well 
Designed 



6% 
2 



10% 
2 



13% 
3 



37% 
3 



5% 
4 



0% 
5 



N= 63 Mean= 2.5 SD= 1.0 



21% 
4 



70% 



Useless Cannot 
Say 



N» 19 Mean= 2.9 SD= 1.3 



10% 
5 : 
Poorly 
Designed 
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GRAPHICS-INPUT 



Comments 



4% 

1 

Valuable 



8% 

1 ; 
Well 
Designed 



8% 
2 



9% 
3 



4% 
4 



3% 
5 



H- 160 Mean= 2.8 SD= 1.2 



26% 
2 



38% 
3 



15% 
4 



72% 



Useless Cannot 
Say 



N= 47 Mean= 3.0 SD=1.1 



13% 
5 : 
Poorly 
Designed 



GRAPHICS- DISPLAY 



5% 

: 1 
Valuable 



16% 
1 

Well 
Designed 



10% 
2 



24% 
2 



8% 
3 



2% 
4 



N= 158 Mean- 2.6 SD= 1.2 



18% 
4 



72% 



Useless Cannot 
Say 



N- 50 Mean= 2.9 SD= 1.3 



14% 
5 : 
Poorly 
Designed 
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Questionnaire for Students who Dropped Course 
Virtual Classroom Project 



Course Name! 

Course Number and Section; 
Instructor: 



Student ID Number: 



X=1.00 SD=0,00 N=9 



SCHOOL 
I am: 

(1) 100% An NJIT Student. 

(2) 0% Upsala Student. 

(3) 0% New School (Connect-Ed) student. 

(4) 0% Other 

drop thrcourLr^^ °^ following factors in your decision to 



Reason 



DHEALTH 

Health problems or 
personal problems 

DHARD 

The course was too hard 
for ne 



Very Somewhat Not X 
Important Important Important 



SD N 



22% 



11% 



78% 2.56 0.88 9 



89% 



2.78 0.67 9 



DWORK 

The course was too much 
work 



11% 



89% 



2.89 0.33 9 



DINSTR 

I did not like the 
instructor 

DBORING 

The subject matter was 
boring or irrelevant 

DDROP 

I had too many other 
courses and needed to 
drop one (or more) 

DPOOR 

I was doing poorly 
DNOLIKE 

I did not like the 
"virtual classroom" 
approach 

DDEMAND 

I hskd too many outside 
demands (other classes, 
full-time work) 
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22% 



22% 



22% 



11% 



22% 



33% 
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22% 



11% 
11% 



56% 2.33 0.87 y 



78% 2.56 0.88 9 



78% 2.56 0.88 9 



78% 2.67 0.71 9 



67% 2,44 0.88 9 



67% 2.33 1.00 9 



DMATCH X«2.44 SD=1.42 N=9 

Tne course aid not match my expectations: 

33% 22% 22% 11% 

J 1 : 2 ! 3 : 4 : 5 



Strongly 
Agree 



Agree 



Don't 
Know 



Disagree 
X"l,56 SD=0.53 N=9 



Strongly 
Disagree 



DTRANS 

I transferred to another 44% Yes 
section of the same 
course 

56% No 

P^^^^N X=3.44 SD-1.59 N=9 

If 1 had the opportunity, I would register for another class which 
used the "Virtual Classroom" approach; 



11% 



Strongly 
Agree 



22% 



Agree 



22% 
3 

Don't 
Know 



0% 



Disagree 



44% 



Strongly 
Disagree 



X=2.62 SD=1.60 N==8 



DMOST 

(1) 38% CONFLICTED 

(2) 12% SIMILAR CLASS 

(3) 12% FAMILY PROBLEMS 

(4) 25% TOO HARD 

(5) 12% DISLIKE INSTRUCTOR 

DBEST X=2.75 SD=0.50 N=4 

What did you like best about the virtual classroom approach? 

(1) 25% IDEOLOGY OF SYSTEM 

(2) 75% CONVENIENCE 

DWORST X«3.00 SD=1.41 N=6 

What did you DISLIKE the most about the virtual classroom as 
implemented in your course? 

(1) 17% LESS TERMINALS 

(2) 17% SYSTEM TOO HARD 

(3) 33% HINDERED DISCUSSION 

(4) 17% ASSIGNMENTS HARD 

(5) 17% DISLIKE INSTRUCTOR 



it was 



ANY ADDITIONAL COMMENTS? 



THANK YOU VEPY MUCH FOR COMPLETING AND 
RETURNING THIS QUESTIONNAIRE TO: 
(USING THE ENCLOSED POSTAGE PAID ENVELOPE) 

Ellen Schreihofer 
CCCC @ NJIT 
32 3 King Blvd. 
Newark, NJ 07102 
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Guide for Personal Interview With Students 



Interviewee^ Date 

Interviewer^^ 

Hello, my name is XX and I am working as (job 
title) in the virtual classroom project. 

What I would like to do is ask you some questions that will give 
us a deeper insight into your own personal experiences and reactions 
to the online course you participated in than we are able to get from 
the standard questionnaire, [if still online... ] Then I would like 
to watch you for a little while while you sign online, and tell me 
what you are thinking as you interact with the system and the class. 

We will share a summary of the comments by all the students in 
^^^^ ^® interview with your instructor, but we will not 
identify any of the comments as coming from any particular student, 

1. Initial recruitment and feelings 

classroom project or 
the experimental online section in wnich you participated? 



What were your initial feelings or reactions... what attracted 
you, what didn't sound good about this approach? 



2. How about the initial training session... after it was over, 
did you feel that you would be able to sign online and find your 
class conference, or was there something than was not clear about 
what the procedure would be?? 



3. Where did you go to use the microcomputer equipment you 
needed to participate each week? 

Were there any problems with the availability of facilities or 
with the lab assistant's ability to help you get online?? 
(probe) 



Did you have any sort of regular schedule each week when you 
would sign online to participate, or how was it that you decided when 
to log on?? 



4. What were your initial feelings or impress ^ns about the 
online class during the training and the first week.' can you 
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remember what you particularly liked, or what you didn't like or 
found confusing? (probe... anything else?) 

r,r.r.^Ju ^ A^^K ^''^^^ ^^^^ rsactlons to reading the comments or 
?h?! ?n^i°"%^'^ students... to What extent did you find 

i-o«^^hi^! 5?^^ ^^t^ .''^''^ P^""^ ^ ^^°"P °^ cla^^^s working 

together, or did you feel that you were pretty much alone in learning 
trie material? ^ 

in ni^5^£n^^ ft^'l u°f 9^°^P)-- Did you or the instructor do anything 
in particular that helped you to be able to work and socialize with 
other students in the online class? 



6' How about the lecture-type material presented by the 

iHf^TlJS . ^° understand the material in 

writing, or do you think you would have learned it better if you had 
listened to it in spoken form? Why?? 



r.Tr.e °^ ^oin any of the public conferences on 

EIES, besides your class conference? 

If yes... which ones, and what did you think of them? 
If no. . . why not?? 



Did you ever exchange messages with anybody online who was not 
in your class or connected to the project? 

If yes... how did this happen? 



How did you feel about this experience of communicating with 
"strangers?" 



8. How would you describe your relationship to your Instructor 
orline.. do you feel MORE or LESS able to communicate and relate to 
your teacher? Why? 
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Q 9 varies depending on whether interview 
is with current student or fall student 
«««« L'^^^^ question for fall students— look first at their 
open-ended questions] 

nu^^°? ^U® 'Juestionnaire you completed, you said that the things you 
liked best were [read quote]. Could you expand on that?? ^ 

^l^^ ^^t^ ^^^^ things you liked least about the virtual 
aiythinrto'I5r?o ^ ' ^-e 

[This question version for current students] 

^ ^ "^Y? developed any particular routines or tricks of the 

^^^^ ^^^^ making EIES more valuable to you than it was at 

first? 



4.U ^ ^^^f ^" online course, what do you like best about 

the virtual classroom approach... that is, what is good about it 

f^I!!?^^^^? a course given in the traditional classroom? [probe. . . 
anything else?] 



What do you currently like least, or feel are the greatest 
problems or shortcomings about this mode of course delivery? 



^ 10. What advice would you give a student who is thinking of 
signing up for an online course? 



How about your instructor.., what advice would you give about 
how they could be more effective if they try teaching this course 
online again/ 



11. Is there anything else you would like to tell us about your 
f^^?^"^."^?""^®^"" anything that was especially funny, or laemorable, or 
valuable, or unpleasant about your experience? 
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Interview 1 

Face-to-Face Interview with a Positive Math 3 05 Student 
Roxanne Hiltz, March 26, 1987 

Q- How did you first hear about the project? 

R- The reason I took it was it was the only section I could get into. 
It said "taught via computer" and I did not know what that 
meant. It fit in my schedule and I had to take something, i 
had never heard about it before— did not know what it was about 
until we got to class the first day. i was real intrigued by 

r. h computers a lot. I've had a lot of fun with EIES. 

Q- What's fun about it? 

A- I don't want to sound bad about the course, but the fun part I've 
had is in the "murder l" conference. That was fun checking it 

group responses were. And the same 
with the statistics class. Before we started going to +quiz and 
+branch, everything was a conference comment. You could see 
what people said, like what they liked and what they didn't 
like... They were putting jokes in there... 

I was trying to think how to describe it when you sent me that 
message, it's DIFFERENT. It's nice to have a class taught a 
different way than everyone sitting in front of the teacher. 
And the teacher goes on and writes, and you write it down... and 
you take the test and hand your homework in.. I do think I miss 
It though because she seems like she'd be GREAT teaching in 
front of a class. I've heard other people who have taken it 
with her there in the class, they say she is a great teacher, 
doesn't bore you or overwhelm you with work. She's kind and 
friendly and everything. 

Q- Does any of her personality come through online? 

R- Yes. she'll put a message in and say, "Have a great week," or 

••Hope your spring break is real nice." Especially, if you have a 
message or a problem, she'll write back, and say, "Hi there, how 

have you been? You have a problem with this " It's r. Lally 

almost like talking on the phone. I try to send message, back 
the same way, real casual. It's not a strict teacher-student 
kind of thing. Because of her, you feel a lot closer, because 
it's so easy just to pop a question. She'll answer the next 
day, or whenever you come online. 

Q- You still feel you have a relationship, though you hardly ever see 
her? 

R- Yes. Also, I won't use the guy's name, but the first class, there 
was this jerk in the back, I said thank God we will not meet in 
this class with him anymore. In a way it's good, I can avoid 
him. . . 

Q- In the beginning it was all comments, trying to get the students 
comfortable with the system and with each other. You're saying 
you were a bit disappointed when it got down to business? 

R" In a way, because you miss participating in a class. But I do like 
the idea, x really do. Like the fact that some people could 
take the class from say, Chicago or California... 

Q- The initial training session... basically, for your class, we did 
not have the right space for it. 

R" yeah, there was a lot of doubling up. I was lucky, I got there 

early and got a terminal. There was a guy who stood next to me, 
and I see him all the time now. We go over the assignments, or 
we yell across the center, "Hey, did you get the homework?" So I 
run over there and get on the system and get it printed out... 
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That's interesting, that you do talk occasionally to other 
students in the class. 
R. Um-humro. And of course you run into them every once in awhile in 
306. It's good for me because it's my first semester here, it's 
almost as easy as meeting somebody in [a regular] class. You 
say, "Hey, I had a problem with number 2, I'll trade with you 
for number 1." For the first half of the semester I was having .^ 
problem doing all of it myself, and there's always some people 
to trade with. ^ ^ 

Q- I'm glad to hear that students still have relationships and work 
on things together... You normally go to 306, you don't have a 
microcomputer? 

R- I have one at home but I don't have a modem. 

Q- How often do you go? 

R- I normally go every day. Except Wednesday, I don't have classes 
then, and it's a long drive. But I go every day about lOam. 
quarter after 10. Print out whatever is on there. 

Q» About how long are your sessions? 

R- Half hour. But that's because I don't sit there and r* .d off the 
computer. I think you'd go blind after awhile! I have it printed 
out and then take it home. Then work out problems and come back 
in the day it's due or the day before. I usually go through it 
once, read the people »s comments and things... I just read them 
once and don't take them home. Then I figure out the lessons and 
whatever and print them out. Then when I come in, I just have to 
log on and put the assignments in. So I'm not on as much as 
somebody who just sits there online and reads it all. But it's 
really better, because for the final or something, what are they 
going to do, read them all again? 

Q- Has there been any problem with having enough printers in 306*' 

R- There's only one that works. The AT&T doesn't have a tractor feed 
and it jams. You just have to time it right, and get thare when 
someone isn't on it. 

Q« Are there any other routines cr ways of using the system that vou 
have (developed that work veil for you? 

R- Not really, because that's really all that I do. 

Q- How do you find the workload compared to other courses? 

R- It's a lot easier, but that's because my other courses, it's 

really wild. This other course, 90% of my time is that course, I 
have to study it all the time, fight my way through it. 
It's like 2- and a half hours a week I spend online, then at 
home, I have to read over the stuff and work on the problems, i 
use a marker on the printouts, and go through the book the same 
wpv. The total time is about five hours a week. But before a 
quxz it's more. Though in essence it's an open book quiz, 
because you have a time limit, if you don't know where to find 
it, it would take more than half an hour to do it. 

Q- Are there any topics in the course where yo;^ found it really hard 
to understand from the combination of what's on line and what's 
in the book? 

R- Yeah, at the beginning, with Uie different kinds of probability 

theory, subjective, a priori, all those... 
Q- so, it wasn't the mathematical part, it was Ihe theoretical or 

philosophical part? 
R. um-humm. I'm not very good at philosophy. The math part, you. have 

v/hat she teaches online and the book md between the two you can 

figure it out. 

Q- What are your general reactions to reading comments by other 
students? 

^ R- They're entertaining, some of those people have some witty 
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comments. That makes the class more interesting, if you find 
that there are a lot of comments, then you get online just to 
see them. I've joined in there and provided a little wit here 
and there. It adds levity, you Jcnow, if everything is all is 
all bare bones and cut and dry, you're not interested, you don't 
want to study that much, you only care about your grade. This 
way, it adds interest to it. 

Q~ Do you feel more or less free to say something witty in a 
conference than you would in a class? 

R- Probably more free. Because, I may seem gregarious, but I'm pretty 
shy. It's easier from here. Because it seems like one-on-one. 

Q~ Do you learn things from the comments of others, or is it more the 
sociable interest? 

R- I think it's more the sociable, in the comments. But we don't have 
the comments now, ve have the branches. 

Q~ How did you get into the murder l experiments... how and when and 
where did you find other things online? 

R- When you first get online, they have an EIES headline, it turned 
out that I was spending a LOT of time at it. There are other 
things on there, but |l,m going to stay away from them the rest 
of the semester. 

Q- Do you now exchange messages with anybody you met in Murder 1' or 
communicate with anybody on the system who's not in '/our class? 

R- Now, I don't. At the end of Murder 1, we were trying to find out 
where we were all from. One guy's in California, Jill's in 
Texas, somebody in Woodbridge. We all said, wouldn't it be 
great if we could get together? But aside from class, I haven't 
gotten together with anybody else on EIES. 
I'm wondering- will I still have that number after this 
semester? 

Interviewer— Yes, evervbody is given the option of keeping their 

number, but it drops to a class two account, which gives poor 

response during busy hours. 
R- Yeah, I'd be curious to. You run into people and you find out they 

are on EIES, And you say, "allright, I'll send you a message." 

Like, what are you doing on Friday? That would happen in any 

Q- You mentioned about quiz and branch. Would you talk about your 
feelings about those procedures? 

R- Plusses- it's a lot harder to cheat. You can't look at conference 
comments and see what everybody else did. Especially when we did 
a ccuple that were actual problems, you wouldn't have to study 
,'f they were reg^ilar comments, I can see the reason for them. 
And I like the idea with Branch that you can find out what other 
people's answers are once you put your answer in. I do know 
people who have abused it. 

Q- How do you abuse it? 

R- You do it with somebr Ay else. And find out the answers. I've seen 
that done. But it's better that you do it alon^. As far as I'm 
concerned, I'm here to learn. Some peoplo are just after -grades. 
There's one guy in a couple of my classes who cheats left and 
right . 

Q- He cheats in all classes? And found a way to do it this way too? 

R- Right. If you want to cheat, you can. You find out w; o those 

people are. And vhen they ask, you say, "Oh, no, I didn't do the 
homework either," you don't give it co them. Bi:ause you don't 
get anything back from them. If it's a two way street, I don't 
consider that cheating. I'm not goxng to say names, but there is 
one person who does not reciprocate, and no one is helping him. 

o No one talks to him in class. 



Q~ This is in the online class? 
R- Both. 

Q- In the beginn.ng, the things that you mentioned that were good 
were that it fit around your schedule, and it's different and 
therefore Interesting. Anything else you can think of as an 
advantage? 

R- Related to the fact that when I signed up for it it was listed as 
Wednesday and Friday, but since it's my only class on Wednesday, 
I don't have to come to school on Wednesday, and I can take it 
whenever... I can take the quiz anytime before Thursday. I find 
it very flexible that way. I can come in between classes and do 
it. So you can "front load" it all in the beginning of the 
week, or wait till the end if something else has come up. That's 
what's good about it. 

Q- What about the greatest problems or shortcomings? 

R- That immediate answer to a problem which you will get in a class. 
Where you raise your hand in class and the teacher answers, a 
lot of times, even it Rose is on, I send the question and sign 
off, because it might be 10 or 15 minutes until she answers. So 
I'll get it the next day. Aside from that, I don't see any 
problems with it. 

Q- What kind of advice would you give a student who sees this on a 
schedule? 

R- I'd say take it, especial if you have not had any computer 

experience. Anybody in their right mind knows that somebody who 
has not had any computer experience will be passed by in jobs by 
somebody who does. 

Q- What you saying is, take it, because besides learning statistics, 
you are going to learn something about computers? 

R- That's one reason. The other is the flexibility. You can learn 
as much as if she were teaching it face to face. 

Q- Do you have any questions fcr me about the project? 

R- Are there plans to have other co rses? 1 think there are 2 or 3 
now? 

Q- Right now we don't necessarily Ir^ave funding for next year. We 
hope to get enough hardware at NJIT so that regularly, there 
might be say half a dozen courses that have online sections. 

R- I'd like to take other courses. There are obviously things it 

would be hard to apply to- say a mechanics course on structures- 
but English 

Interviewer — Yes, we had an English course online. And CIS seems to 
work well. 

R- Yes, that would. But unfortunately, I took 213 already. I'd like 
to take another course. 

[ end of interview chat and thanks edited out] 
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Interview 2 

TELEPHONE INTERVIEW WITH NEGATIVE MATH 305 STUDENT 

George Baldwin, 7/2/87 

Q: DO you remember what it was like when you first came into the 
program? That xs, when they told you it was to be offered 
"online" how did you feel? 

A: I didn't like iti 

Q: Did it come as a surprise to you? 

A: Yes 

Q: Did they offer you an alternative? 

A: Yes, a night class. And that didn't fit my schedule. 
Q: So, did you feel that they were not being straightforward with 
you about how the class was to be offered? 

nil' ^i«^Sf"I^ ^^t registration material, if that's what you 

mean. Maybe it was, but I just missed it. 
Q: They gave you a training session. How did you feel about the 

training conference? 
A: Just fine. There were plenty of people to help. 
Q: Where did you usually use the computer to get online? 
A: At my home. I have a computer at home. 

^' 1^^^^^^°^^^ ^^^^ was convenient for you to do your 

A: Once I got a modem, yeh! 

Q: Did they explain to you how to plug the modem into your 
computer? 

A: I already knew how to do that. I usually signed on when I got 
none at night. 

A: since you were signing on from home, there weren't any lab 

assistants available to you. Did you have any problems in not 
naving h^lp? 

A: I could usually hack my way around! 

Q: Did you have a regular schedule that you followed each week in 
signing on for class? 

A: No.... That was one of my biggest problems...! know you're going 
to ask that. Personally, I usually try not to take self-study 
classes. And that's pretty much what that is. I don't feel 
that I have the self discipline for it. I don't have enough 
time in my day as it is. To sit down and make myself do 

something like that Self study takes a certain kind of 

person to do that. And I'm not that kind of person. 

Q: The first week that you signed on to the system... can you 
remember what your first impressions of the system were? 

A: It didn't bother me to take a class online like that. I am not 
scared of computers. I just don't like self study. I don't 
have complaints about the class. 

Q: When you did go online with the other students, how did you feel 
about the comments and contributions of the other class members? 
Were they of any use to you, or did you find them to be just a 
waste of time? 

A: I usually just blew-off the other class members comments and 
went straight to the professors lectures. I wouldn't say that 
the other students comments were a waste of my time, I just 
didn't read theml 
Q: Did you feel like you were part of a group, or were you pretty 
much alone in learning the material? 

Pretty much alone but I feci that way about my other classes 

as well. I have two other friends that I sometimes study with, 
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but that's about it. 

your^!jork?^^"^'^^°'' anything that you found helpful in doing 

^' S^L^'^M®?^ helpful. She gave us her home phone number and we 

o. n?r^on^! 1° call her at the of f ice. .. .voice or online. 

Q. Did you ever ask her for help online? 

A: No, usually i went to see her at her office. 

Q: How did you feel about the lecture material presented by vour 
wrftSf ?oL^^S^/"' ^^^^ ^^^^^"9 lectures 

voice lectSe? ^""^^ preferred a real-time 

Ai I would have rather had it in a regular lecture type classroom. 

fJ-ZL ""rt^^^ i^^l fi^^^"^ ^" ^ classroom, but I find ireas!;r 
ror me. It works better, if you have a question you can iust 
stop and ask right then and there. ^ ^ 

Q: While you were doing your online class, did you ever join any 
other public conferences? ^ 

A: No. 

Q: Why? 

A: I worked full time and went to school full time. Busy, it's 
klnda tough. I'd usually go to work... then to school then to 

^""f^ study at 11 at night, and I 

aian t want to sit down and read some other stuff. 

private messages? Did you private message any of the 
other people in your class? ' y °"y »^ ^ne 

A: Yes. 

Q: How often, and why? 

A: Sometimes to ask them a question about something-or-another. 
Some of the time with the T A. . .who was there to help, i send 
her messages sometimes. 

^" P?IvatrmeSsa5e:?°"''''^ °' 

^' ?H^«J tt^^'r ^"°!r ?J?^°"® outside of the class. I really don't 
fvJ" , r?^r,-^ system as well as I could have. But I 

think that's because I didn't have any time. 

Q: The relationship that you had with your teacher— Did you feel 
Soriputer?^^° ^^^^ ^° communicate with the professor through the 

A: Less 



QJ You mentioned that you liked to go to her office and talk with 
her. • 

A: Yes, more personal. 

Q: You wrote on your questionnaire that you felt that the EIES 
system was impersonal. That's one of the things I'd like to 
talk about. could you tell me why you thought it was 
impersonal? 

A: Because you had a computer in front of you. .. instead of a 

?»f^°?:4 i^^^ i ^^^^ before, if i have a question in class I can 
just stick my hand up. You can't do that on a computer. You 
just have that text in front of you. You know how it 
is. ..sometimes you can sidetrack the professor. Get 'em to bull 
«hit for a while... you can't do that on a computer. 

Q: What was the bfist thing about the EIES system? 

A: I could sit down and do my work at any time. 

Q: And the worst thing? 

A: That it was self study. If it came down to taking another 

S^i Z IJ'^^u''^!''?.^^^® again, 'I'd really have to think long 

and hard about it. I usually try not to. 

Q: If you had a friend or another student who asked you for your 
advice about taking an online class, what would you tell them? 

A39 2 79 



A: I'd tell them what it was like, then say "It's up to you as to 
whether you can do it or not." It's self study. 

Qj If you were talking with a professor who was thinking about 
teaching an online class, what advice would you give the prof? 

A: I'd tell 'era that you have to make yourselt avai;iable. You 
really do. Personal interaction. 

Q: Not just online? 

A: Yeh. You don't have to have millions of office hours, but you 
gotta make yourself a little bit more available. Just in case 
there is a problem. i .you just can't settle it over a computer. 

■klfkifk 

[Note: Besides regular office hours, the instructor made herself 
available in the Lab for two hours every Thursday, specifically 
to help online students with any difficulties. This student 
shows no recognition of these opportunities.] 
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Inten/iew 3 

INTERVIEW WITH MODERATELY POSITIVE CC140Y STUDENT 
>! 4.K . . Roxanne Hiltz, April lo, 1987 

"ck"?^''"'*^*^ "'^ssag^ requesting volunteers for 

interviews. She is one of the better students in the course. 

Key: I or Q = Remark or question from Roxanne Hiltz, Interviewer 
R - response irom student 

Q- HOW did you first hear about the experimental means of delivery 
for this course? «xxv«iy 

^' class, the first day. i had no idea before then. 

Q- What were your impressions that first day? 

!!oI;:^!!?i:'" tjj^^hs...) -^hat are we getting into... this is 
something new I" 

Q. Did you consider transferring to another section at that point? 
wo, not me. (Q- why not?) R- Because I knew that every new 
thing, in the beginning it's hard, but then you get used to it, 

Q- So, to you It was sort of a challenge? 

R"" Yes* 

Q. Do you have any thoughts about those first couple of sessions 
iny ways they should have been done better? ' 

R- The handouts were very good. I used them later on. In the first 
session, I was really confused, I did not know what was going 
on. It weis pretty crowded; hard to listen and hard to know 2hat 
was going on. I paid attention but it did not help too much, i 
think it would be better in a classroom. Put It or the board and 
go over the handouts. 

Q- one idea would be to start in the lab; do a very brief demo, 
saying "here is the system you are going to use;" move to a 
classroom and explain it on the board, referring to the 
handouts ... 

R- Yes, and then go back to the lab afterwards. some people may 
not have used the handouts. y ^ "lay 

Interviewer... maybe that is why they had so much trouble... 
I think it is better If the teacher himself give the lecture. 
Show about the conference, where to put messages.. 

Q- Try to compare this to courses you have had in the traditional 
classroom... what do you like, what don't you like, what 
problems you have had. . . 

R- What's good is that you don't have to be in class. But that's a 
problem, too. I'll say, ' I have time, till Wednesday,' then 
once you start to read everything you find out you have to read 
it at least twice. You sometimes don't do your homework really 
well, because it is a last-moment exercise, before you know it." 
If you have class every two days, you know you have to be 
prepared. So, it's good and bad... you have the time, but you 
have to know how to make good use of it. 

Another problem is, sometimes you don't feel comfortable asking 
the teacher questions through the computer. In class, you can 
raise your hand, or you can ask questions after class. It is not 
as comfortable to ask a question online, so you don't ask. 
Probe- what makes you feel less comfortable? 

R" maybe he will take off credits or something. Sometimes it is too 
late to put a question in (the assignment is already due). It's 
more personal when you see the teacher. 

Q- Do you ever work at all with other students in the class? 
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R- Not really working, but sometimes pcopXe ask questions. 

Q- How could things be improved sio that there is wore cammunication 
with the teacher and the other students? 

R- With the teacher, che communication was good, it was just the 
part about asking questions about the material. The idea of 
quizzes- probably nobody will like it, but X think it was; a good 
idea. It will force you to sit down an do your homework. You 
have to do it on time, which is very good. He should start this 
from the beginning, what has happened is, now I got my new 
assignments, and so many things were put at the last moment. He 
should put more in the beginning, to divide it better so all the 
pressure does not come at the end. 

How to improve... hmm. . . it's cute idea to use it; I like it. 
Maybe a few more examples would help. 

Maybe he should direct, "this student and this student, you have 
to do this assignment together, through the computer." 

Q- Ok... one of the things we wanted to do was somehow getting the 
students doing something together online. 

R- From the beginning to the end, this could be done, but the 
partners could be changed for different assignments. 

Q- How would you feel about being assigned a partner? 

R- No problem. Like (name) and I, we write each other messages all 
the time about questions. But nobody else asks me questions 
other than (X) . So, you could change partners, and sometimes 
have a group assignment. Then, you do have to depend on your 
partner; and if he doesn't do his part in the end... 

Q- Yes, that is one problem, assigiiing grades when the partners do 
not work equally. That's a good idea, though. 

Q- Do you think that the freshman or sophomore level is good for 
this sort of course delivery, or should it be only fc upoer 
level courses? 

R- It you want to study, you study; it does not matter if you are a 

Freshman or a Junior or a Senior, it's a challenge. 
Q- Is there anything else you particularly liked about this mode of 

delivery, other than not having to go to class? 
R- Yes, i like having the complete lecture. You can get it and 

then read it three days later; or you can go in in the middle of 

the night. It was easier for me. 
Q- Some of the students have not been very active in the class at 

all. Have any of them said anything to you about why this is? 
R- Probably because they don't have the time. They postpone it 

until the last minute, and then the last minute it is too late 

to do everything. 
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Interview 4 

INFORMAL INTERVIEW AND OBSERVATIONS OF NEGATIVE CC140y STUDENT 

Roxanne Hiltz, March 26, 1987 
*. >* ^^^^ office at Upsala, a disgruntled virtual classroom 

student appeared. He was very impatient about being helped by a lab 

f«22S\/T^^''®?.*'?^?'^®f exiting the Lab, muttering epithets, 
221^?^^^ I complained that he "had to use th4 

IZf^lt ^5® lab assistant had asked him to wait a few 

minutes while she finished helping another student. He claimed that 
he had only a few minutes to "get the stuff." Auamea tnat 

This student is very unhappy with the online section and with 
the course, which is computer-assisted statistics, required for 
freshmen as the second half of a sequence of courses which introduces 
them to computer usage. He said he wanted to transfer to a different 
section; but, he could not take any other sections scheduled because 
they conflict with his other obligations. Moreover, he does not like 
computers and does not want to have anything to do with them. 

The student is very negative towards any interaction with other 
students, or anything beyond the minimum necessary to get through 
this required course. In tne first assignment, where he was 
instructed to enter a conference comment answering a survey, so that 
the students could use each other's answers to compute some 
statistics on the data, he had been instructed to use the key 
"survey." He added the key, "junk." He feels that all of the 
material students are entering is "junk," 

*. i.uf print all his waiting conference comments, about 26 

at this point. When I came back, I noted that he had ripped off the 
ones entered by the instructor, but left behind as unwanted co/.nments 
entered by other students. He had not signed on to enter anything 
himself. He had not entered anything himself, just ripped off what 
he wanted and left. 
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Interview 5 

TSL^lPHONE INTERVIEW WITH NEGATIVE CC140 STUDENT 
George Baldwin, 6/30/87 

In the follow up questionnaire, this student indicated that she did 
1 ? classroom approach. By her own admission she 

also did not do well with her final grade, she DID like the 
instructor. Text of the interview follows. 

Q- Do you have time to talk with me about your experiences with 

your online EIES computer class? 

A- I have time but I didn't do well in the class! 

^" ^Jl^^'^s o^ay. I just want to find out about your experience 

A- Oh, okay! 

Q- Great, po you remember what it was like the first time you used 

the terminal to go to class? 
A- I thought it was interesting. 

Q- You weren't intimidated by the machine or the computer at all"? 
A- Oh, no. Because before I had taken Stats I had had computers.* 
Q- Then you had used computers before. How did you feel about the 
course being taught online— could you give me your impressions 
about how you felt when you heard that the course would be 
offered online? 

^* understand it at first, i thought that, it , well, I 

didn't really know what it meant, i didn't know what "online" 

meant until I got there. 
Q- They gave you some training before you went online. Do you 

remember what the training was like and how you felt about it' 
A- Well, there were two training sessions and they were 2 hours 

each and they went over all the commands and how to hook up to 

EIE£ and the phone numbers. 
Q- Did you feel confident in signing on, and getting into your 

class conference? 
A- Well, it took me about a week more before I could sign on and 

get on by myself. They had people there to help me in the 

beginning, but then I started picking up on my own. 
Q- Where did you go to get online? That is, where did you use a 

computer for signing onto your class conference? 
A- At my college— at Upsala. 

Q- You didn't have any problems of access to a computer? 

A- Well, no. Sometimes they were busy, but I usually didn't have 

to wait too long. .. .about 5 minutes. 
Q- Did you have problems in getting help from a lab assistant? 
A- No. They were the ones who helped me get online in the 

beginning and then I picked it up. And they helped me print It 

out . 

Q- How often do you recall signing on each week? 

A- My course was only six weeks: m the beginning I didn't sign on 

at all because I was sick. Then I started going three times a 

week. Same as my other classes. 
Q- Oh, then you had a regular time for signina on? 
A- Yes. About 10:00. It fit into my other classes. 
Q- If you had a computer at home, would you have signed on from 

there or still gone to school? 
A- I probably would have blown off the one at school ... after I had 

learned to sign on at the lab.... maybe getting my assignments at 

school, but doing my work at home. 
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^ SS^4.^f,^„ !^ ^^f ^^^^^ ^^9"®^ can you remember 

what you particularly liked or disliked about the EXES system? 
A- Well, I thought it was pretty easy to understand, i mean, the 
computer told you everything to do! The only thing that I had 
trouble with was like trying to get to the scratchpad. . .because 
like in the beginning you had to hit break and all that. 
Q- Did you have problems with the scratchpad editor? 
A- No, not at all. 

Q- When other students made entries, and you read them... how did 
you find their comments; useful, interesting, or a waste of 
time? 

A- Well, most of the students who made comments were the ones who 
really understood the class and they were about the lectures. 
And they were pretty helpful, especially when the homework could 
be checked. 

Q" Did you make very many comments yourself? 
A- No. 

A- You had to send conferences to your teacher, but I didn't sav 
much . ' 

Q- Well tell me: Did you feel that you were a part of a class, or 

did you feel alone in your studies? 
A- Well, we had a buddy system, and this other girl and I went 

together. I felt by myself, or with her. 
Q- About the instructor. Was the instructor there to help you? 

Q- Your instructor presented lecture material online. Did you find 
It easier to understand it in writing, or would you have 
preferred to have had it in a verbal type lecture? 
A- I liked it better in writing. 
Q- Better than in a spoken form? Why was that? 
A- Because I always find that when someone gives me something 
written I find it easier to comprehend it. Sometimes when 
people talk your mind wanders and you don't get it. But when 
it's written, its all there for you. When people talk they 
don't keep repeating themselves. This way I could just read it 
over and understand it. 
Q- Do you think you keep good notes in class? 

A- Sometimes, but like I say my riind wanders and you just try to 
pick It up but it's pretty hard. But otherwise I take decent 
notes . 

Q« While you were on EIES, did you try any of the public 

conferences besides the one for your class? 
A- Well, my girlfriend was in a French class and she connected with 
some- one from NJIT and they conversed back and forth. And I 
used to read some of her stuff about what they would say to each 
other. 

Q- Why did you not join any of the other conferences? were you 
short on time, or did you just not know about the public 
conferences? 

A- T guess I just didn't know about them. The only conferences I 

read were the ones with my class. 
Q- Did your teacher, or any instructors ever mention to you about 

the other public conferences? 
A- Some lady did mention that you could get messages or conferences 

somewhere around the third week. 
Q- Did you ever exchange messages with someone not in your class? 
A- I n«iver exchanged messages with someone not in my class, but I 
did with people in my class. Like when I got sick, I left 
messages for that girl I used to go with. She got it okayl 
Q- Could you describe your relationship with your instructor lor 
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me? Were you able to communicate well with him? 
A- Well, I did find it hard if I had a problem, because I was sick. 
?" i.^?!*"^ ^"^^^^^ because you didn't have a terminal at home, or 
Si said he was always in his office, and everything. 

^ point, could you tell me about what you liked about the 

EIES Virtual classroom approach? What do you think was the best 

aspect of it? 

A- I liked that i was indepenendent and that I could go whenever I 
wanted to. And I liked how the conferences were written down 
and I could get my notes, it also helps if you miss a day or 
two. Because the computer always has your assignments there for 

Q- What did you like the least about it? 

A- There wasn't anything that I didn't really like about it. i 

thought it was very well organized. 
Q- If another student came up to you and said they were thinking 

about signing up for an online course, what kind of advice would 

you give that student? 
A- Not to get sick! Unless they have a computer at home. 
Q- I am thinking about teaching a course online. Speaking to me as 

a teacher, what advice would you give ma about teaching an 

online course? ^ 

A- Well, the teacher that I had was very good. He left messages 
about what time he would be available for students, if you had 
any questions or problems. And in his lectures he would put the 
pages and the chapters. And you could read them along with his 
lectures if you didn't understand it. 
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Interview 6 
INTERVIEW WITH SATISFIED OSS VC STUDENT 
Roxanne Hiltz, March 29, 1987 

Q- I want you to think back to when you first heard about how this 
course was going to be delivered.,, what were your impress ions'? 

R- It was the first day of class. I liked it. I thought it was 
cocl. I just like working with the computer, it seemed 
different... fun. it's turned out pretty good so far. 

Q- What is it that it about it that is fun? 

"^fj? °^ cL^as time. You're not pressured to have 

something done right away- you can do it whenever you want. You 
can be more free on the computer. Some kids are hesitant to 
speak up in class. You can put in your thoughts., people read 
it. Sometimes they care, sometimes they don't. If they don't 
they can just skip on by it. it's also good because there is 
easy access whenever you want. I have a modem at home. I can go 
on at 3 o'clock in the morning. That's usually when I do most 
of my work. 

P- Do you do it all at home, or do you sometimes use the computers 
here? 

R- I never use the school system. I heard a lot of kids have 
problems in the library. They leave messages like, "sorry, I 
can't read what I«m writing." I bought a modem specially for the 
class. I saw that it would be kind of useless to come down 
every day to use the school system. It works much better from 
home , 

P- Anything else that you like about it? 

R- I don't know. Dr. Hsu makes all the students participate. For 
instance, we have two businesses competing against each other, 
and separate conferences to compote. The more work you do and 
the better your profits, the better your grade. So that's pretty 
good. If you do the work, you get the grade. He does not base 
It so much on book learning. I can't learn it from a book. You 
learn it as you go along, you get ideas from everybody else, not 
what one author thinks. 

P- Picking up on that- to what extent do you think it is either 
boring or interesting to read comments from the other students? 

R- Dr. Hsu really doesn't say all that much. He leaves it almost 
entirely up to the kids. I think it's good. I'm learning so 
many things. We're not held tight to OS471. I've learned a lot, 
like people ask about jobs, and just general information giving. 
For instance, we have a chapter on informal and formal 
businesses, and he'll say, "What's your opinion?" Then you can 
read other people's opinions and let that help you form your 
opinion. You can go back and change your opinion if you want, 
which is good, 

Q- Do you send private messages to anybody? 

R" Yeah, I have a few people" I send messages to. (gives names) 

(some are on staff) . 
P- so sometimes it's about the course, and sometimes it's not? 
R- yeah, sometimes it's just goofing around. 

Q- have you gotten involved in anything else 9oing on on EXES? 

R- Not really. I've played around a little. Nothing seemed to 
concern me to much. I was looking at INterests; I saw other 
people had them listed, but I couldn't figure it out. And the 
games. The reviews and surveys I haven't figured out how to 
work yet. But i got time. 
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Interviewer- survey, you should reall^' stay out of... if you want 

n ^ f etjap an online survey, it's pretty complicated. 

Q- What about your relationship with Dr. Hsu? Does he seem the 

same online, or different online? 
R- He's pretty much the same online. I think he's a really good 

««oh i;?^i^^^°''"?*f"^ ^^"^ highly. I think that JoSrse is 

much better with the online part, i think you'd be bored to 
death sitting there learning about business every time. The 
application is great. I'm glad that I have it. I'd take it 
??/ ® ®^5®a<*y recommended it to my friends, i think 

it's just so much better. And Dr. Hsu, he's great. He's got a 
great personality, not a real strict or stingy type guy. For the 
l^f^ SSf^ wants your opinion on things, and he'll conform to 
y°^L°P f??^' ^'^^^^ how we want him to grade our 

testL.. He leaves things up to us. He has a good sense of humor 
Hoo • 

disadvantages? Do you think you're missing anything 
« S f^*?®^?.^^^"^® ^^^^ about using the system? 

Definitely disadvantages. Some people are lazy, i just can't 
afford to be lazy because that's what the teacher is grading you 
on. But online you don't have to log on if you don't want to. 
You're only hurting yourself.... No, you're hurting the Sthe?s, 
too, because they aren^t getting your opinion on things. I guess 
some of them are restricted to school use, and the computers 
u^®4.^J!® 1^ ^^■^^ ^^^^ anyway. You don't have to worry 
about keeping records because you can always call everything 
back up. I don't see too many disadvantages really, I've never 
had a problem with it. jr x v« never 

Q- have you used branching? 
R- yes 

Q- what was your reaction to that? 

R- I'm not crazy about it. It's kind of a hassle to put in, and 

vm1^'r.^?n°?^^^';°''^^ branches. I see why he does it. But once 
you re in it, you see everything anyway, so what's the 
difference? You could just put it on the regular conference. 
It s for organization type purposes I guess. For instance, our 
resumes are all in a branch. I never went back once I put mine 
in, personally, so I don't see too great an advantage of it, 
except for organization. ' 

Q~ have you developed some sort of routine; ways of attacking the 
system once you sign on? 

R- yeah, I always do the same thing. I always want to see who's 
online, in case any of my friends aie online. Then, if there 
are any messages, I'll read them. Then I'll either answer the 
messages if there were any left for me; then I'll go to 
conference choice. I always do conference 1732 first; th^n the 
other one for the company, i read all the response??, i save 
everything to disk, so I just usually scroll it up. Then I'll go 
back and writ© comments. Then I goof around after that, and log 
off. ^ 

Q- How much time do you spend? 

R" I sign on every day. I usually spend about an hour; it depends 
how much other work I have. Sometimes as little as half an 
hour; sometimes two or three hours. Sometimes I sign on several 
times a day. I spend a lot of time online. I love it. 

Q- More than you spend on other courses? 

R- yeah. This is my favorite course. I don't mind putting in the 
hours, the time just flies by. i> ^ 

Q- Suppose we were doing a brochure for prospective students. Are 
there any earnings we should give them? what kind of advice 



would you give them? 
R- If you do it from home, make sure you contact the telephone 
company first because you can run up a big phone bill if you 
don't get special services from them. I called about a week' 
after I got my first phone bill. Now it just costs about $4.00 a 
month. 

P- We did mention this the first class., but you think we should be 

stronger about this. 
R- yeah. I was running up 20 to 30 hours a month. (tape ran out 

here and fumble- fingers interviewer took awhile to get new one 

in) 

Q- If you were going to add something to what's available, can you 
think of any featuras that you would like to have available? 

R- Not now. I'll have to think about that. That's a tough one. 
I've never used any other system. 

Q- anything you would like to anonymously Dr. Hsu? 

R- He has a group read a chapter and put ti* Ir outline on the 
system, i think it's a wast of time because no one is reading 
the chapters. Nobody asks questions in class because nobody's 
rading the chapters. When it came time for the midterm, we were 
all complaining. Plus his two points for every right answer and 
two points off for every wrong one- it's different. That didn't 
go over too well. I like the way he presents his material. 
Sometimes he puts a comment in and says you have until that 
night to do ot- who has free time like that? I'm not too crazy 
about that. But 1 guess he has to do it that way and it's our 
fault for not getting it on. 

Q- Can you think of anything else that was especially memorable, or 
funny? 

R- On the system. Sure, when we were all first trying to learn it, 
sending messages to each other, some of them were going to the 
wrong people. Someone in class got an anonymous message This guy 
Stacey got a message from "an admirer," and it was anonymous. 
And no one wanted to claim it. 

F- I see. The man's name was Stacy; so it could be a woman's name? 
Somebody sent it to him and looked again- and oops! 

R- Yes. he left a note on the conference saying, thanks, \o whoever 
sent him the message, and he hopes he gets another one. No one 
has since written a message. Then the teacher. Dr. Hsu, put on a 
quest ivon, "what's your opinion on anonymous messages?" Tha**. was 
pretty funny. And his making it an assignment, that was good. 
He does that a lot, asking your personal opionion on everything. 
That was pretty funny. 

Q- Do you have anything you want to know about the project or the 
system? 

R- what's coming now? I know there's going to be a new EIES? 
(interviewer explains 2 new systems and prototypes) We hope 
eventually to put up a utility and to offer a number of courses 
this way. 

R- So you wouldn't have to come to class at all? I think that 

would really be something to try to see how i**. works. I would do 
that« interviwer- most of the courses online now are totally 
online. 

R- Oh, are they? I didn't know that. 

Interviewer- they come to their training, and then don't meet again 
until the final exam. 

R- Yeah, we tried to coax our teacher to give us an online midterm. 
He would't go for that. We asked if we could have a virtual 
midterm, and he said, "Hov» would you like a virtual grade?" 
Q Interviewer- The problem is that it is posisible to cheat. 



R- he mentioned that you were thinking of having a time limit and 
you couldn't log off until it's done, that'a a pretty good 
solution. Of course it's still possible to cheat, unless you 
make it explicitly an open-book, open-notes exam. (Interviewer 
explains that 213 and NJIT statistics course are totally online 
this semester) . 

R- too bad i had 213 already. 

Q- Anything else you can think of? 

R- Yeah, what courses are you going to be putting online? 
Interviewer- I don't know. Right now we don't have a mainframe to 

regularly run on. 
R- what about keeping our account? 
Interviewer explains... 
R- that's good. 

(more irrelevant chat about future plans) 
Thanks a lot for stopping by. 
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Interview 7 

Telephone Interview with VC Dropout Not Responding to Questionnaire 

Conducted by George Baldwin 
Class: OS471-Management Lab- NJIT 

Q: (first name) , I am doing a follow-up study on students who 

dropped out of the virtual classroom project. (Name), you did 
a management lab with us here at NJIT. Would you raina 
talking with.i me a few moments about why you dropped the 
course? It's kind of important to us. 

A: I don't have time to get down there, I'm on a tight 
schedule. I used to go to work right after clasps. X used to 
have to go there and put time in on that terminal. 
....I was never able to do it. 

Q: So basically it was coming over to NJIT to use the tfjrminal? 

A: I would already be at NJIT. I, like I said, run a tight 
schedule. After my classes, I scheduled so I could go right to 
work. And I don't have a modem at home. 

Q: So do you think it would have been a little easier if you had a 
modem at home? 

A: Yeh. And they didn't really go over the right procedures for 
getting a modem or getting an outside phone line to hook it up. 
They didn't go over that at all. 

Q: So you would have appreciated it if 

A: Yeh: I had to drop that class, and I wouldn't have had to take 
it over again during my senior summer which is what I am going 
to have to do now if I want to graduate in May. 

Qi Oh. 

A: I would have rather had taken a different course if I didn't 
have to take that one. I wouldn't have had to drop it. 
Thats the reason I dropped it. I couldn't do those stupid 
classrooms. I couldn't get the lab time in. Thats why I 
dropped the course. 

Q: How did you feel about the Virtual Classroom approach? 

A: Ah, I really couldn't, I really can't evaluate it. I wasn't in 
It long enough, ya know. 

Q: Ok. 

A: I only went to the first couple of EIES sessions. I never even 

got my thing, my resume into the class. 
Q: How about the subject matter of the course? Was it boring or 

relevant to you? You mentioned that you would have rather had 

another class. 

A: I would have rather taken another OS471 class without the 

computer because then I wouldn't have had to drop. And I would 
have had the course. 

Q: Did you ever get to meet the instructor? 

A: Ah, which one? My teacher? 

Q: Yes. Did you like him? 

A: Yeh, he's alright. Dr. Huang— H-Z-U. 

Q: So what you are going to do this summer is retake it? During 

your senior summer? 
A: Well, I'm a senior now and I'll graduate with my class in 

May. I will need six credits to graduate. That's what I mean 

by my senior summer. 

Q: Tell me (Name) .. .would you register for another class using the 

VC approach? 
A: It depends. I can't answer that question. 

Q: If you had a modem at home and a computer 
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A: If I had it already, yeh. 

Q: Did you ever sign-on to the class? (Yeh) Is there anything 
that you particularly like about the VC approach. 

A: Gee, its been so long... I liked the way you could send 
messages. ..How everyone was tied in together like that. 

Q: Do you renenber what it was like the first time you signed on 
to the system? ^ 

A: 1 didn't think it was any big deal. 

Q: Had you used computers before? 

A: Yeh! I have one at home! 

Q: Do you recall anything that you particularly disliked about 
the VC approach, at least as it was used in the course that vou 
were taking? ^ 

A: I really don't have an opinion, one way or the other. I only 
went to the classroom twice! At that point I was just 
running around and falling more and more behind in that 
class. And that's why I decided to drop it. I stopped going to 
it and I dropped it really late. It might look like I stayed in 
it longer than I did, but I stopped going because I realized 
that I didn't have the time for it. I dropped it at the end of 
the year. 
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Interview 8 

Face-to-Face Interview with a Moderately Positive Student 
Organizational Communication Course 
April, 1987 Starr Roxanne Hiltz 

Note: This student was not among the best in the course, either in 
terms of grade (B) or subjective ratings. However, the student 
did like the system well enough to request continuation of an 
account beyond the expiriation of the course. 

Int: When you first heard that part of this course would occur 
online, in a "virtual classroom," what was your reaction? 

R:^ I heard about it through the professor in the course. I didn't 
know what it was all about, what it would be, except that it 
would involve a computer... In the beginning, it was really 
hard, for some of the kids in the class. I myself adjusted ok. 

Int: Where did you go to participate, and did you have any problems 
with equipment? 

R: I went either at night, or right before or after class. See, 
our class was half online and half off. The problem I had was, 
there was a terminal in the library, but it did not wo:'- half of 
the time. 

Int: How did you feel about reading the comments of other students? 
Did you find it valuable to read their opinions, or did you 
think that they were not worth reading? 

R: I felt that they were really helpful. It gave me another 

perspective on what I was doing. If I did not see a point and 
they did, I was able to incorporate it into my thinking. As far 
as me responding, it gave me a chance to read what the other 
students wrote. Sometimes they responded and sometimes they 
cUdn't. It was a really good way of learning different ideas. 

Int: When you were online communicating with your instructor (who 

hcppned to be me!), did you feel that you were more able or less 
able to communicate? 

R: About the same. Except that it was especially good the time when 
you were away in Florida, this is the only way we could have 
communicated with you. If we had not had EIES, we could never 
have talked with you about the problems we were having with our 
work for the course, it was a good tool. Now, I still have an 
account, and if J need to get hold of one of my professors who 
has an account, then they will get it and respond. It is an 
easier way to get hold of a professor. 

Int: When you were taking part of your course online, what things did 
you dislike the most, or find annoying? And on the other hand, 
what aspects of using the system did you find the most valuable? 

R: Well, a lot of times online, things can be taken the wrong way. 
If you write a message and you do not put it in exactly the 
right way, somebody would get upset. It's not what you meant, 
but they took it the wrong way. Also, if you were in a hurry, 
you could not just sign on, quickly leave a message, and sign 
off again, because it was so slow. If I tried it in between 
classes, i would be late. Communicating one-to-one, face to 
face is quicker, but you can't always find the person. Also, 
I'm a commuter, and you can't make a trip to campus just to 
leave a message. 

Int: So, your problems were basically with the mode of 

communication— being careful lest you be misunderstood— and 
with access and response, a few weeks ago. Something came off 
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so^ii*''^*''*^ way. It's hard. You have to write everything "just 

Int: There is something called "flaming" online, where people get 

upset, and because they cannot see one another, it can escalate. 
Rj Yes (laughs) . 

Int. You mentioned ssome problems. Is there any way in which this 
approach is better than face-to-face class meetings? 
well. If everybody signs on and gives a response, then everybody 
wL^'^i^^^? ""^"^l everybody else is thinking, it's an easier 

way to get a class together. But I still prefer one-on-one. 
It's hard talking through a machine. 

Int: If you were given a choice of a class totally online or totally 
face-to-face, then, you would take the face-to-face class? 

R: I'd rather have it half and half. That way, you could have the 
experience of going online, but also be able to talk to the 
professor. 

Int! Is there anything else you can think of in terms of things you 
learned from your experience? 

R: la lot about the computer, about conferencing... I learned a 
lot. Having the experience with telecommunications, it sparked 
something... i want to go into that. I don't know, I fell in 
love with the computer, even though it was hard sometimes. 

Int! could you explain about that? You said that it was harder, that 
it was colder, yet you say you "fell in love with the computer?" 
I ^2?'^ Jcnow, it's just that communicating like that, it's like 
talking on the telephone, only you're typing in. It's just 
amazing. It's colder but it's amazing at the same time, it's 
really interesting. I just like it a lot... it was wierd, here 
you were meeting people. You don't know them, but all of a 
sudden you get a message from them. 

Int! So, other people sent you messages? 

R! Yes, I never started it, but other people sent me messages, I 

guess because I was female. And I met a whole different group of 
people. People from Texas, from Florida, it's incredible. 

Int! Your reactions were not totally typical. There were several 
students in the class who really disliked it. [R: i know, I 
know. ] There were one or two who say they dropped the course 
because they could not stand the computer. 

R! I can't believe that. 

Int! Do you have any observations or interpretation related to this? 
Why their reactions were so negative, and whether anything could 
have been done? 

Ri People tend to be scared of the computer, and to get really 

frustrated. [When they have trouble, they say] Right away, "oh 
my God! I don't know any of this! Why should I be bothered?" 
They just give up. I think if you stick with something, I think 
they would have learned to like it. Maybe if they had just tried 
harder, they would have liked it„ 

Int: Anything else you can remember about any problems? 

R! Yes, with the microlab. With the lab assistants. Sometimes, 

they were there, but they were playing games or whatever. When 
you said, "i need help," they wouldn't move, or not quickly 
enough . 

Int! So you're saying they were not actively helpful, they seemed to 

be preoccupied. 
R! Yes, they were passive. 

Int: Were there any assistants who were particularly good? 
R: Yes, (name) and (name) were very good. 
Int! What is it that (name) and (name) did that was good? 
cD?r> Well, (name) knew ALL the tricks, and he was there to help. If 



you said, "I really need your help," he would get up and coine 

over right away, and he knew v^hat to do. 
Ints So, its actively helping the students and being knowledgable, 

not just being there physically? 
R: yes. 

Ints Thank you very much [etc etc] 
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Interview 9 

TELEPHONE INTERVIEW WITH A NEGATIVE CIS STUDENT 
Starr Roxanne Hiltz, August 2, 1987 

Q: How did you first hear about the virtual classroom project or the 
Online CIS 213 course? 

R: I read about it in the registration material, and I decided i did 
not want to take it. Being a [non-CIS] major and going to 
school at night, I worked full time during the day, and I didn't 
know whether I would be able to dedicate enough time to it. 
What happened was, when I went to register, I registered for 
another course. And the registrar said there was a course open 
on Wednesday night, so I registered for it. The night the 
course was starting, that's when I found out it was the virtual 
classroom. 

Q: I see, so you really got into it by accident... 

R: He saw two CIS 213 's, and he put me in the one that was open. 

Q: What were your initial feelings at that point, when you got to the 
training and discovered what you were in? 

R: I had a lot of apprehension. I mentioned it to the instructor, 
BJ, that I was kind of worried. I was ready to dedicate the 
time that I could to it, but I was still worried, because I 
thought there was a little extra that was needed. I think BJ 
mentioned that if there was any way he could help, he would do 
his best to assist. So I said, OK, how bad can it be? And 
besides, I really needed to take the course, and I wanted to 
take the course, and so I decided to stick with it. I was a 
little nervous. I really didn't want to take a virtual 
classroom course. 

Q: Where did you go to use a microcomputer? Did you have equipment 
at home, or go to the Lab, or what? 

R: That was the other problem. I felt that I was at a disadvantage 
because I did not have a personal computer. I really didn't 
have access to a computer except at work, and I had to compete 
against everyone else. There were like four for a department of 
30 people, each running their own LOTUS or whatever; I had to 
fit myself between there. So I was forc«d to work after hours, 
7-8 o'clock at night. Even that was a problem, because then I 
had to battle against the guys that were cleaning and waxing. I 
was at the point where I was getting frustrated because I was 
not able to work on it when I wanted to. The days when I really 
wanted to work on it were Saturdays and Sundays., I do a lot of 
my homework then. And the Institute was not open. I guess it 
was open, if you made an appointment or something you could get 
in for a fm hours. 

Q: So it would have been much easier for you if you could have gone 
somewhere on weekends. 

R: Either that, or I was even looking for a good way to lease a 
personal computer. I went to a couple of places after the 
semester had started, and prices were just oatrageous. To rent 
a computer for a month, it was like $400.00, 

Q: Yes, that really is outrageous, they don't sell for a whole lot 
more than that! 

R: I was really tight for money at that time. XJndvr the 
circumstances, I just had to clo the best I could. 

Q: So, you work full time, and you would normally go in about one 
night a week? 

R: That's about all I had. That's exactly what it wcrked out to be. 
I was able to dedicate one full night a week. That was like five 
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or six hours a week. On the initial questionnaire, it asked 
about how much time you would spend, and I felt I was going to 
need at least two to three hours a day. If l had my own 
computer... but I didn't, I wasn't able to dedicate that much 
time. I think I did get a D in that course, i was hoping for 
something better, but under the circumstances, I think that a D 
was probably the most appropriate grade. ^ i wish I had done 
better and I think I would have been able to. 

Q: That long session one night a week, did you do it from work or did 
you go into NJIT? 

R: Prom work, and I tried to get into NJIT a few times, when I did 
that, it was only to try to clean up the bugs on a program, it 
was more desperation, get-it-done kind of work, and I really was 
not picking up a whole lot of information... 

Q: When you were out there, fighting for a computer at work, did you 
feel that you were part of a group or class, or did you feel 
really all alone out there. 

R: That's a good question. I always knew that I was part of a group, 
but I also felt alone because I did not communicate like I 
really wanted to. There were some of the students who were on, 
sending messages two or three times a day. Those are the ones 
who communicated with each other. I even tried to do it, but 
with once a week, I didn't get a whole lot of practice with 
sending messages. For instance, I sent a few to [name], but it 
was difficult, because he had already progressed to a point 
where he could send messages all the time. I could send a 
regular message, but I didn't know about how to to the talking 
online... the ones who were doing that were the ones who were on 
two or three times a day. I couldn't devote that much time to 
it, and as a result, I didn't have that much communication. And 
as a result of that, I was more of an individual rather than 
part of a group. 

Q: Did you read the comments by other students in the conference, or 
did you tend to skip over them? 

R: My downfall was in trying to mimimize reading of the comments 

during the time I had to devote to it. I didn't read them on the 
screen, I printed them out and took th?m home. Then things 
would happen. I work long hours, I live alone and have to cook 
dinner... I did look at a few of them, but my downfall was that 
I tried to do everything as fast as I could in order to maximize 
what I could finish during that one night. I tried to bring the 
papen^^ork home, but you bring home a book and often it does not 
happen... i read maybe 60% of it. There were things that I 
would look for, like the lectures. Then I would make my own 
little notes on the print of the lectures. 

Q: On the lectures, do you find it easier or harder to understand the 
material in writing, as compared to hearing it? 

R: That was easy. They were clear and well put together... 

Sometimes I had some trouble associating the material in the 
lecture with what was in the book. If you didn't do that, and 
then tried to walk in and take the quiz., it became more and 
more difficult. 

Q: Did you ever send messages to BJ, or didn't you have time to do 
that? 

R: I sent a few, but not nearly as much as I really wanted to. I 

think BJ noticed that right off the bat, he oven mentioned that 
I was getting kind of quiet. I did send messages to him, things 
like whether I could get more time. We talked, but I don't think 
I got to be as close to him as I wanted to. 
Thinking about the relationship you had with BJ online, and the 



y??'''® ^^^^ professors whom you have seen one 
with BJ? Did you feel closer or further away? 

«i"i^^^^^^°^®5'.*'''^ ^ ^^^^ ^ have gotten as close as some 

getting. That was one of ?Se 
problems... looking at some of the coinments other students were 

^^.''J? ^T'^^ questions he's asking, 

thougS?s ?ike ?ha?r ^^^^^"^ ^ ^^^^^"^ 

Q: Well, because you were not there every day, right? 

"vour^^Hv?L^''''>,\°S?''2^ i^^! ^^^^ registration and asked for 
your advice, what kind of advice would you give? 

R: It would depend on the individual. I could not make a 

recommendation to a person I don't know. I could recommend it 

!?»f°!®2"fu^ h^^® ^i"^®' and will dedicate the 

time and the effort, and who has the the equipment available at 

«n!!!So««^»? ^^\u^ considered that probably the key to being 

nonSf! class. Especially if the person needed the 

5 otherwise have to wait another semester; then 

I would recommend it. 

^' ^"work'^harder?^^ equipment, and if they are willing to 

R: Well, I don't know if you have to work harder, but if you don't 
have the computer at home, you work 9 or 10 hours a day, then 
you have to go home and cook dinner... somebody like that, 
somebody in exactly the same circumstances that I had, I would 
have to think very, very hard about it. You want to get the 
J??^,^?? °^ * course, and I don't think I did. I don't 

tS Sork with.^'"* ''^^ ^^""^"^ ®^iP"»ent 

Q: What about BJ? What advice would you like to pass on to him if he 
were ever to do this again? ^ 

R: There were a couple of things... I don't know if I conveyed them 
iS^^S® questionnaire, when I filled it out it was like the night 
?u ?® final, and I just wanted to get it done and study for 

the final. I wanted to do more "dinky" programs. I think we did 
two big programs that were assigned to us. I would have loved 
to have seen a tiny little program due every week, along with a 
couple of big ones. The practice would have been good, I would 
have become more familiar with it. 

Int: Well, J really appreciate your taking the time to talk to me. 
I WxSh things had worked out better for you. 

R: Believe it or not, I just bought another book on Pascal 

programming. I still want to learn. I'm still trying to pick it 

?!1 ^ when the course ended, and I really 

didn't pick up as much as I wanted to. 
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Interview 10 



Telephone Interview with Posit ivfe CIS213 Student 
Interviewer : Roxanne Hiltz- July 1987 



I- Hi I (name). This is Roxanne. 
R- Ok. 
I 



R 
I- 



wia? ! lound w?n^;%^ ''ii^ be anonymously using some quotes. 

^11 Lf through the data from the survey is that 

Jh«? ?^ I^ind of dry. So I'd like to ask you some cjuestiSns so 

that I can get more of a feeling for what you really experienced. 

R- Ok. 

I- I want you first to think all the way back to last January when 
?earl 5^?^^''^ about the VC project for CIS213 online l^ Shat yoS 

p ^ reactions were and why you decided to take it! 
i? ?n^fS«" it was over the December vacation that I 
got it in the mail and I saw that they were offering CIS213 - 
(this is for me you know since I already knew Pascal) - I saw it 
?r ??^t ^ ^ and everything, so I said "Easy 
^L'i/S^S^ really why I wanted to take it! I knew pretty much^ 
about modems and all that stuff, and I said it should be pretty 
easy. You know you got to keep grades pretty 

I- Okay. So you had a modem and Microcomputer right at home and 
access wasn't any problem for you? y au norae ana 

R- Yeah! It was at school - I was on campus. 

I- You were on campus. So it was like in your dorm room? 
K- Yean! 

l" Sioh^?!J«?*^?.!?^ technical difficulties at anytime? 

dilficSmes I didn't have any kind of 

you actually got started what were your feelinas at that 
f^was ^'Lfv^.J°^ initially like? Dislike? Ld you sUii ?hink^' 
What? °^ ^^^^"^ ^^i'^^^ happening or 

^" !!^: ^ ^^^^^ thought it was an easy "A". I thought the system 
nnS*"*;*. ^ ^°"'t - I couldn't tell you how 

^rnnnH k^^^^""' ^"^^ ^^^"^^ was pretty easj to get 

n^H V? "k^ "^^"^"^ ^®*i problems - No major problems. 

si«n«H on?r of regular schedule each week when you were 

take par?? "'^^^ ^^^'^^^ t° participate or 

R- Ahh! It was like whenever I had spare time. 

^" loMJSnY.^/^"^;?'^ ^"^^ - like when I wanted 

^ho SJ^^i? ^^^^f ''^^ d^"P it in my buffer like 2 in 

next da?!"^"* whenever I had the chance and I'd print it out 

you^leisS^f ^'^^"^ ""^^ ^° download and then print and read it at 

^" ""^-^^ ^i^^t. But just to have a hard copy, you 

^S^n review ana an mat, it was a lot easier than going 
thru . . . you know ... ^ ^ 

^' ^TUS Sk"*!?®!?^® that other students made— to what extent did you 
find fhat it was interesting or helpful or engaging to see some 

o?i2r«5,^S"^ 7.°'' to what extent did you think that what the 

other students said was a waste of time? 
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R- I'm not sure I understand you You said that some people said 

- "it was a waste of time"? 

I- Nol Some of the comments were entered by the instructor and some 
of the material was students making contributions, making comments 
or asking questions. Did you like to read the other students 
comments, or skip over them, or what was your reaction to things 
done by other students? 

!t- No, I thought, you know, it was helpful to have everybody 

participate - I didn't skip over - I read everytning ~ just to see 
-you know, compare yourself to the rest of the class and know how 
they are doing and what they know and what they don't know and all 
that. 

I- When you were out there at the end of your modem, did you feel 
that you were part of a class or a group that was working 
together, or did you feel you were kindda all alone? 

R- No....Nahl I felt that I was computing a little bit. No! I thought 
I was in a classroom, you know. 

I- Do you think that (instructor's name) did anything that was 

helpful to the class to become like a class that worked together, 
or on the other hand that he hindered that? 

R- Did (instructor's name) ? 

I- Is there anything that he did that helped -the class to feel like a 
real group, or that got in the way of students feeling like it was 
a real class? 

R-The Instructor tried to get everyone involved. He posted comments 
to encourage participation. 

I- What about lecture type material? comparing reading it to 
listening to it in a classroom. What do you see are the 
advantages/disadvantages? 

R- May not want to be in class at the scheduled time - So it is 

better online. In addition, the lecture was only outlining major 
points covered. Advantages to both offline and online. For me it 
was easier in online lecture. The instructor made up the lecture 
personally, not as if he were copying it out of a book. Made it 
easy. 

I- What about other people on EXES. Did you ever get involved in 
conferences or messages with people outside your class? 

R- Yes, I joined a few public and private conferences and made 
friends that I still have now. 

I- Wow I That is interesting. Are they friends in New Jersey? 

R- Some people in my class and others - I even have mv own conference 
now. Turbo pascal conference. 

I- Oh! I didn't know that. So you went from student to conference 
moderator. How would you describe your relationship with 
(instructor's name) as an instructor? Did you feel more able or 
less able to relate to him as compared to teachtirn* in a regular 
classroom? 

R- Little more able tc relate. 

I- Why? 

R- Because he doesn't s«*g you - all he knows is what you type. He 

can't be prejudiced against you based on the way you look. 
I- So you are raying he was more objective in this medium in reacting 

to the work you did and not to other things? 
R- Yes! It's more fair this way. You're being judged really on your 

work, not on your personality. 
I- Do you think you learned more/ less or the same as you would have 

if you took this course in a regular classroom? 
R~ More for me - Learned the system - I already knew Pascal. Most 

people who take this course already know a little about computers. 
J- So you're saying that you didn't necessarily learn mors course 
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SompSteh? • ""^"^^ "t'^^'^ "S^s °* 

yS^Zu^' people aren't en .^us^ • tie about computers - I don't know 

I- Why do you think they woulci probably do less? 

R- Well! Most people are intimidated about it, wouldn't understand it 
- Its kindda like a hassle when you are first learning it, 
especially for those people who have not used computers since thev 
were young like I did. So I got the hang of it qSickly. ^ 

" di^SSvKe'L'tKir'iSnr ' '"'^'^ " 

I- Yes. Besides the fact that people who aren't familiar with 

r^n^von'^fKY^v ^f''" ^^^^'^ ^^^y hurdle - 

Can you think of other problems or shortcomings in trying to use a 
Virtual Classroom approach to deliver courses? 
R- For me? No! I didn't have any problems. 

^" ?!JoV \l ^ ^^"'^?2*'t°?'"? "?.^° s^^^' I saw this thing - you 

«?So?^l:/5?"i^ ^^^^^ °^ '^ind of advice would you 

give? What kinds of questions would you ask, to think about in 
trying tv decide if the student should do this or a regular 
course? ^ 

R- I'd say definitely do it. 

I- For everybody or just certain kinds of students? 

^" ^'^ probably say for everybody - I'm not saying they'd get 

as good a grade - I'm just saying, do it for the eJpe^ience and 
lor tun. 

I- Tell me about that. What was especially fun about it or more 

memorable - what made it different and fun? 
R- I don't know - Just a change - Seems like more fun. 
I- Do you think it was fun because it was the first time you did it - 

I mean if you took another course, would that be fun too, or wouid 

it not be as much fun? 
R- I don't know - I've used a lot of computer systems, and for me it 

IS just more convenient/interesting. 
I- Okay. Is there anything else you can think about? It's been a 

Z u^^fi® ^^^^ h^^® clone differently or better? 

What should people know about in order to understand what it's 
like to take a course this way? 

- Nol Not really. 

- Okay. Thank you. Goodnight. 
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